» Blog » Networking » Network Routing Problems

I’m currently under the impression that most ISPs in the Seattle-Tacoma region do not have direct peering with one-another at all. While this is a small sample of tests, I do find it strikingly odd that almost every single test within these two cities and between these two cities always routes through San Jose, California.

My four local test ISPs are
1) Click-Network Cable (Tacoma, WA)
2) Comcast Business Cable (Puyallup, WA)
3) Optic Fusion (Tacoma, WA)
4) Hurricane Electric (Seattle, WA)

And for added out-of-state testing, I’m including
5) Hurricane Electric (Fremont CA)

NOTE: These tests are all performed using tracert (Windows).
NOTE: some DNS names and IP addresses were removed to protect the identity of Darkain Multimedia’s tech support clients.

Round 1 – Click-Network (Tacoma WA)

Test 1-2: Click-Network to Comcast Business

  1    <1 ms    <1 ms    <1 ms  192.168.10.1
  2    15 ms     8 ms     7 ms  10.102.0.1
  3    14 ms     9 ms     9 ms  131.191.0.14
  4    10 ms    11 ms     9 ms  click_dts_border_g1-23.click-network.com [131.191.0.26]
  5     9 ms     9 ms    10 ms  sl-gw20-tac-10-0.sprintlink.net [144.228.24.133]
  6    12 ms    13 ms    13 ms  sl-crs1-tac-0-9-3-0.sprintlink.net [144.232.11.59]
  7    10 ms     9 ms     9 ms  sl-gw50-sea-.sprintlink.net [144.232.1.151]
  8    12 ms    16 ms    19 ms  te-0-5-0-2-pe03.seattle.wa.ibone.comcast.net [66.208.228.77]
  9    12 ms    14 ms    11 ms  be-13-cr01.seattle.wa.ibone.comcast.net [68.86.84.109]
 10    25 ms    14 ms    15 ms  so-6-1-0-0-ar03.seattle.wa.seattle.comcast.net [68.86.95.22]
 11    12 ms    13 ms    14 ms  po-60-ur02.tacoma.wa.seattle.comcast.net [68.85.240.90]
 12    12 ms    14 ms    14 ms  68.87.207.234
 13    25 ms    24 ms    26 ms  xxxxxx.hsd1.wa.comcast.net [xxx.xxx.xxx.xxx]

This first test worked perfectly. The routing path of Tacoma WA - Seattle WA - Tacoma WA - Puyallup WA is perfectly acceptable. Seattle WA is a much larger carrier exchange than Tacoma WA, so these results are expected.

Test 1-3: Click-Network to Optic Fusion

  1    <1 ms    <1 ms    <1 ms  192.168.10.1
  2     8 ms     8 ms     7 ms  10.102.0.1
  3     9 ms     9 ms     9 ms  131.191.0.14
  4     8 ms     9 ms    10 ms  click_dts_border_g1-23.click-network.com [131.191.0.26]
  5    15 ms     9 ms    10 ms  sl-gw20-tac-10-2.sprintlink.net [144.223.84.169]
  6    21 ms    13 ms     9 ms  sl-crs2-tac-0-9-3-0.sprintlink.net [144.232.11.61]
  7    33 ms    33 ms    30 ms  sl-crs2-sj-0-5-2-0.sprintlink.net [144.232.18.204]
  8    34 ms    35 ms    33 ms  sl-st31-sj-0-15-0-0.sprintlink.net [144.232.8.151]
  9    32 ms    42 ms    40 ms  sl-integ92-316012-0.sprintlink.net [144.223.166.114]
 10    61 ms    55 ms    54 ms  tg9-3.cr02.sntdcabl.integra.net [209.63.81.69]
 11    53 ms    53 ms    51 ms  tg13-1.cr02.rcrdcauu.integra.net [209.63.114.169]
 12    50 ms    52 ms    51 ms  209.63.83.9
 13    86 ms    55 ms    50 ms  209.63.83.5
 14    51 ms    52 ms    54 ms  ge5-1.ar10.ptleorte.integra.net [209.63.115.18]
 15   121 ms    54 ms    53 ms  gi2-2.cr01.tac.opticfusion.net [64.122.170.154]
 16    80 ms    73 ms    60 ms  gi2-1.dr01.tac.opticfusion.net [209.147.112.46]
 17    65 ms    58 ms    54 ms  xxx.xxx.xxx.xxx

This next test is the first indicator of a problem. Optic Fusion has carrier exchanges in both Tacoma WA and Seattle WA. Click-Network routes through SprintLink, which from the first trace route, we observe that SprinkLink has direct routing to the carrier exchange in Seattle as well. Despite both of these potential links, this particular trace runs the route of Tacoma WA - San Jose CA - Portland OR - Tacoma WA.

What makes this one much worse though... Optic Fusion lists Click-Network as a direct peering partner. This means that Click-Network should in theory be routing packets directly to Optic Fusion for this router, completely bypassing SprintLink altogether. (source: http://www.opticfusion.net/netmon/weathermap.png)

Test 1-4: Click-Network to Hurricane Electric (Seattle WA)

  1     4 ms     3 ms    <1 ms  192.168.10.1
  2     7 ms     7 ms    16 ms  10.102.0.1
  3     8 ms     8 ms     8 ms  131.191.0.16
  4    11 ms    14 ms    69 ms  click_dtn_border_g1-23.click-network.com [131.191.0.24]
  5     9 ms     8 ms     8 ms  sl-gw5-tac-4-1.sprintlink.net [144.223.158.229]
  6    26 ms    32 ms    31 ms  sl-gw5-tac-14-0.sprintlink.net [144.232.2.2]
  7    33 ms    33 ms    41 ms  sl-crs2-sj-0-5-2-0.sprintlink.net [144.232.18.204]
  8    36 ms    34 ms    32 ms  sl-st31-sj-0-15-0-0.sprintlink.net [144.232.8.151]
  9    33 ms    31 ms    45 ms  144.232.18.142
 10    33 ms    32 ms    41 ms  hurricane-ic-138359-sjo-bb1.c.telia.net [213.248.67.106]
 11    54 ms    53 ms    52 ms  10gigabitethernet4-3.core1.sea1.he.net [72.52.92.158]
 12    52 ms    53 ms    51 ms  tserv14.sea1.ipv6.he.net [216.218.226.238]

As we see once again, SprintLink has opted to route through San Jose CA despite having direct access to Seattle WA. From San Jose CA, Hurricane Electric uses a direct link to bring the route back up to Seattle WA. That makes this full route as Tacoma WA - San Jose CA - Seattle WA.

Test 1-5: Click-Network to Hurricane Electric (Fremont CA)

  1    <1 ms    <1 ms    <1 ms  192.168.10.1
  2     7 ms     6 ms     7 ms  10.102.0.1
  3     9 ms     8 ms     9 ms  131.191.0.16
  4    10 ms    11 ms    10 ms  131.191.0.32
  5    16 ms    23 ms    20 ms  sl-gw5-tac-4-1.sprintlink.net [144.223.158.229]
  6     9 ms     9 ms    11 ms  sl-gw5-tac-14-0.sprintlink.net [144.232.2.2]
  7    35 ms    34 ms    32 ms  sl-crs2-sj-0-5-2-0.sprintlink.net [144.232.18.204]
  8    32 ms    32 ms    35 ms  sl-st31-sj-0-15-0-0.sprintlink.net [144.232.8.151]
  9    33 ms    38 ms    32 ms  144.232.18.142
 10    44 ms    34 ms    38 ms  hurricane-ic-138359-sjo-bb1.c.telia.net [213.248.67.106]
 11    36 ms    86 ms    36 ms  10gigabitethernet1-2.core1.fmt2.he.net [72.52.92.73]
 12    38 ms    34 ms    33 ms  tserv3.fmt2.ipv6.he.net [72.52.104.74]

The two Hurricane Electric IP addresses used for testing are both IPv6 Tunnel Broker addresses (see tunnelbroker.net for more details). I assumed when signing up for the IPv6 tunnel broker service that using either Optic Fusion or Hurricane Electric (Seattle WA) would be my lowest latency solution. After countless testing and retesting, the results presented here are the norm. Routing to Hurricane Electric (Fremont CA) from Click-Network (Tacoma WA) has consistently lower latency than routing through Optic Fusion (Tacoma WA) or Hurricane Electric (Seattle WA).

Round 2 - Comcast Business (Puyallup WA)

Test 2-1: Comcast Business to Click-Network

  1    <1 ms    <1 ms    <1 ms  192.168.100.1
  2     1 ms    <1 ms    <1 ms  10.1.10.1
  3    11 ms    11 ms    10 ms  73.102.132.1
  4    10 ms    10 ms    10 ms  te-4-5-ur02.tacoma.wa.seattle.comcast.net [68.87.207.233]
  5    12 ms    11 ms    13 ms  ae-6-0-ar03.seattle.wa.seattle.comcast.net [68.85.240.89]
  6    16 ms    16 ms    16 ms  pos-1-14-0-0-cr01.seattle.wa.ibone.comcast.net [68.86.90.85]
  7    16 ms    11 ms    13 ms  be-11-pe03.seattle.wa.ibone.comcast.net [68.86.84.78]
  8    12 ms    16 ms    15 ms  66.208.228.78
  9    68 ms    21 ms    35 ms  sl-crs1-tac-.sprintlink.net [144.232.1.149]
 10    18 ms    18 ms    17 ms  sl-gw20-tac-0-0-0.sprintlink.net [144.232.11.58]
 11    16 ms    18 ms    17 ms  sl-click1-418525-0.sprintlink.net [144.228.24.134]
 12    18 ms    16 ms    17 ms  click_dts_core_g1-23.click-network.com [131.191.0.27]
 13    22 ms    19 ms    21 ms  131.191.0.15

Much like with the previous test, the reverse path is pretty much the same. Puyallup WA - Tacoma WA - Seattle WA - Tacoma WA

Test 2-3: Comcast Business to Optic Fusion

  1    <1 ms    <1 ms     1 ms  192.168.100.1
  2    <1 ms    <1 ms    <1 ms  10.1.10.1
  3    10 ms    10 ms    17 ms  73.102.132.1
  4    59 ms    15 ms    19 ms  te-4-5-ur02.tacoma.wa.seattle.comcast.net [68.87.207.233]
  5    11 ms    13 ms    67 ms  ae-6-0-ar03.seattle.wa.seattle.comcast.net [68.85.240.89]
  6    17 ms    15 ms    17 ms  pos-1-14-0-0-cr01.seattle.wa.ibone.comcast.net [68.86.90.85]
  7    14 ms    14 ms    12 ms  pos-0-0-0-0-pe01.seattle.wa.ibone.comcast.net [68.86.86.138]
  8    14 ms    13 ms    16 ms  173-167-56-190-static.hfc.comcastbusiness.net [173.167.56.190]
  9    92 ms   218 ms   212 ms  tg9-4.cr01.sttlwatw.integra.net [209.63.114.133]
 10    19 ms    21 ms    23 ms  pc1--1001.br01.lsancarc.integra.net [209.63.114.149]
 11    21 ms    20 ms    19 ms  tg13-1.cr01.ptleorte.integra.net [209.63.114.98]
 12    98 ms    21 ms    18 ms  ge5-1.ar10.ptleorte.integra.net [209.63.115.18]
 13    24 ms    24 ms    24 ms  gi2-2.cr01.tac.opticfusion.net [64.122.170.154]
 14    22 ms    21 ms    23 ms  209.147.112.50
 15    72 ms   167 ms   199 ms  vl100.dr01.tac.opticfusion.net [66.113.96.1]
 16    26 ms    23 ms    24 ms  xxx.xxx.xxx.xxx

While not as bad as the previous ISP's results, this one still raises some questions. Once again Optic Fusion lists Comcast on their "connectivity" page (source: http://www.opticfusion.net/connectivity.php). This route traces through Puyallup WA - Tacoma WA - Seattle WA - Portland OR - Tacoma WA. Once again we see where in Seattle WA both Comcast and Optic Fusion have carrier exchanges, however they are not directly peering with one-another.

Test 2-4: Comcast Business to Hurricane Electric (Seattle WA)

  1    <1 ms    <1 ms    <1 ms  192.168.100.1
  2    15 ms    <1 ms    <1 ms  10.1.10.1
  3    11 ms    17 ms    12 ms  73.102.132.1
  4    12 ms    16 ms    19 ms  te-7-5-ur01.tacoma.wa.seattle.comcast.net [68.87.207.225]
  5    13 ms    11 ms    11 ms  ae-6-0-ar03.burien.wa.seattle.comcast.net [68.85.240.85]
  6    15 ms    48 ms    12 ms  ae-2-0-ar03.seattle.wa.seattle.comcast.net [68.86.177.146]
  7    14 ms    15 ms    14 ms  pos-1-4-0-0-cr01.seattle.wa.ibone.comcast.net [68.86.90.209]
  8   136 ms   226 ms   203 ms  208.178.58.85
  9    42 ms    51 ms    34 ms  Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net [64.214.174.246]
 10    31 ms    43 ms    71 ms  10gigabitethernet4-3.core1.sea1.he.net [72.52.92.158]
 11    33 ms    33 ms   100 ms  tserv14.sea1.ipv6.he.net [216.218.226.238]

Just like with the previous ISP, Comcast is routing through San Jose to reach Hurricane Electric (Seattle WA). This time it routes through Puyallup WA - Tacoma WA - Burien WA - Seattle WA - San Jose CA - Seattle WA. Yet another missed carrier exchange in Seattle WA!

Test 2-5: Comcast Business to Hurricane Electric (Fremont CA)

  1    <1 ms    <1 ms    <1 ms  192.168.100.1
  2     1 ms    <1 ms     2 ms  10.1.10.1
  3    11 ms     9 ms     7 ms  73.102.132.1
  4     9 ms     9 ms    19 ms  te-7-5-ur01.tacoma.wa.seattle.comcast.net [68.87.207.225]
  5    13 ms    10 ms     9 ms  ae-6-0-ar03.burien.wa.seattle.comcast.net [68.85.240.85]
  6    22 ms     9 ms    18 ms  ae-2-0-ar03.seattle.wa.seattle.comcast.net [68.86.177.146]
  7    15 ms    16 ms    16 ms  pos-1-4-0-0-cr01.seattle.wa.ibone.comcast.net [68.86.90.209]
  8    95 ms    14 ms    12 ms  208.178.58.85
  9    42 ms    37 ms    35 ms  Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net [64.214.174.246]
 10    45 ms    45 ms    48 ms  10gigabitethernet1-2.core1.fmt2.he.net [72.52.92.73]
 11    39 ms    39 ms    38 ms  tserv3.fmt2.ipv6.he.net [72.52.104.74]

Once again we're routing through San Jose CA. Unlike the previous ISP though, both Fremont CA and Seattle WA endpoints for Hurricane Electric are running at about the same latency on this path. This route traces through Puyallup WA - Tacoma WA - Burien WA - Seattle WA - San Jose CA - Seattle WA.

Round 3 - Optic Fusion (Tacoma WA)

Test 3-1: Optic Fusion to Click-Network

  1     9 ms    <1 ms    <1 ms  172.30.0.1
  2     1 ms     7 ms    <1 ms  fa7-16.dr01.tac.opticfusion.net [209.147.125.209]
  3     1 ms    <1 ms    <1 ms  gi2-4.cr02.tac.opticfusion.net [209.147.112.6]
  4     3 ms     3 ms     2 ms  gi2-7.cr02.sea.opticfusion.net [209.147.112.54]
  5     3 ms     3 ms     2 ms  64.125.186.33.mpr2.sea1.us.above.net [64.125.186.33]
  6    22 ms    21 ms    21 ms  xe-1-2-0.cr2.sjc2.us.above.net [64.125.31.22]
  7    22 ms    46 ms    21 ms  xe-0-0-0.cr1.sjc2.us.above.net [64.125.30.125]
  8    22 ms    21 ms    21 ms  xe-4-3-0.er1.sjc2.us.above.net [64.125.28.53]
  9    22 ms    21 ms    21 ms  xe-0-1-0.mpr3.sjc7.us.above.net [64.125.30.174]
 10    22 ms    22 ms    22 ms  sl-st20-sj-11-0-4.sprintlink.net [144.228.111.89]
 11    24 ms    23 ms    23 ms  sl-crs2-sj-0-2-0-0.sprintlink.net [144.232.9.59]
 12    46 ms    47 ms    46 ms  sl-crs2-tac-0-5-3-0.sprintlink.net [144.232.18.205]
 13    45 ms    45 ms    45 ms  sl-gw20-tac-9-0-0.sprintlink.net [144.232.11.60]
 14    46 ms    46 ms    45 ms  sl-click1-388001-0.sprintlink.net [144.223.158.14]
 15    46 ms    45 ms    45 ms  click_dts_core_g1-23.click-network.com [131.191.0.27]
 16    46 ms    46 ms    46 ms  131.191.0.15

Here we can see another reverse path. This time it routes through Above.net instead of Integra. This route traces through Tacoma WA - Seattle WA - San Jose CA - Tacoma WA. And once again we have missed potential carrier exchanges in both Tacoma WA and Seattle WA.

Test 3-2: Optic Fusion to Comcast Business

  1     1 ms    <1 ms    <1 ms  172.30.0.1
  2     1 ms    <1 ms    <1 ms  fa7-16.dr01.tac.opticfusion.net [209.147.125.209]
  3     2 ms     1 ms     1 ms  gi2-4.cr01.tac.opticfusion.net [209.147.112.45]
  4     5 ms     4 ms    11 ms  64.122.170.153
  5    11 ms    10 ms    11 ms  ge2-10.cr01.ptleorte.integra.net [209.63.115.17]
  6    12 ms    15 ms    15 ms  tg13-1.cr01.sttlwatw.integra.net [209.63.114.97]
  7    11 ms    11 ms    11 ms  bs01.lsanca54.integra.net [209.63.114.150]
  8    11 ms    10 ms    11 ms  tg1-1.br01.sttlwawb.integra.net [209.63.114.134]
  9    12 ms    11 ms    11 ms  be-10-206-pe01.seattle.wa.ibone.comcast.net [173.167.56.189]
 10    13 ms    14 ms    15 ms  pos-0-5-0-0-cr01.seattle.wa.ibone.comcast.net [68.86.85.37]
 11    11 ms    11 ms    11 ms  so-1-1-0-0-ar03.seattle.wa.seattle.comcast.net [68.86.93.102]
 12    14 ms    14 ms    13 ms  po-60-ur02.tacoma.wa.seattle.comcast.net [68.85.240.90]
 13    14 ms    14 ms    13 ms  68.87.207.234
 14    21 ms    21 ms    22 ms  xxxxxx.hsd1.wa.comcast.net [xxx.xxx.xxx.xxx]

Much like with the reverse path, we see this route pass through Portland when Optic Fusion a direct carrier exchange in Seattle WA. The full route is Tacoma WA - Portland OR - Seattle WA - Tacoma WA - Puyallup WA

Test 3-4: Optic Fusion to Hurricane Electric (Seattle WA)

  1    <1 ms    <1 ms    <1 ms  172.30.0.1
  2     1 ms    <1 ms     1 ms  fa7-16.dr01.tac.opticfusion.net [209.147.125.209]
  3     1 ms     1 ms    <1 ms  gi2-4.cr02.tac.opticfusion.net [209.147.112.6]
  4     3 ms     3 ms     2 ms  gi2-7.cr02.sea.opticfusion.net [209.147.112.54]
  5     3 ms     2 ms     2 ms  64.125.186.33.mpr2.sea1.us.above.net [64.125.186.33]
  6     3 ms     3 ms     3 ms  xe-0-0-0.mpr1.sea1.us.above.net [64.125.31.9]
  7     3 ms     3 ms     3 ms  10gigabitethernet1-3.core1.sea1.he.net [206.81.80.40]
  8     3 ms     3 ms     2 ms  tserv14.sea1.ipv6.he.net [216.218.226.238]

I think we finally have a winner here! This route is great, it is using only local carrier exchanges. Optic Fusion uses their own internal hop from Tacoma WA to Seattle WA followed by a near direct connection to Hurricane Electric from there. This really shows what the ISPs could have if they routed properly: ~2-3 millisecond latency from start-point to end-point. The entire route is simply Tacoma WA to Seattle WA as expected! This may be a result of both points being data centers rather than consumer/business end points.

Test 3-5: Optic Fusion to Hurricane Electric (Fremont CA)

  1    <1 ms    <1 ms    <1 ms  172.30.0.1
  2     1 ms    <1 ms    <1 ms  fa7-16.dr01.tac.opticfusion.net [209.147.125.209]
  3     1 ms    <1 ms    <1 ms  gi2-4.cr02.tac.opticfusion.net [209.147.112.6]
  4     3 ms     3 ms     2 ms  gi2-7.cr02.sea.opticfusion.net [209.147.112.54]
  5     3 ms     2 ms     2 ms  64.125.186.33.mpr2.sea1.us.above.net [64.125.186.33]
  6     3 ms     3 ms     2 ms  xe-0-0-0.mpr1.sea1.us.above.net [64.125.31.9]
  7     3 ms     3 ms     3 ms  10gigabitethernet1-3.core1.sea1.he.net [206.81.80.40]
  8    30 ms    24 ms    23 ms  10gigabitethernet9-1.core1.sjc2.he.net [72.52.92.157]
  9    24 ms    24 ms    24 ms  10gigabitethernet1-2.core1.fmt2.he.net [72.52.92.73]
 10    24 ms    24 ms    24 ms  tserv3.fmt2.ipv6.he.net [72.52.104.74]

Once again we're seeing almost direct routing to the end-point. This follows the same path as the previous trace to their Seattle WA end point and then proceeds to follow Hurricane Electric's network south to their Fremont CA end point. The full route is Tacoma WA - Seattle WA - San Jose CA - Fremont CA.

Final Thoughts

First, neither of Hurricane Electric end points were tested as starting points. I do not personally have that sort of access to their networks to test this out.

Next, it appears as though if you are not a data center attempting to route to another geographical close data center then your routes are going to be on higher latency lengthy paths. For my particular purposes of IPv6 tunneling over IPv4 it looks as though I should use Fremont CA (~33ms at 700 miles) instead of Tacoma WA (~55ms at 3 miles) or Seattle WA (~55ms at 35 miles).

This entry was posted in Networking. Bookmark the permalink.