ReefCentral

[MalHavoc]

Originally Posted by nickb

I just tried a tracert to Reefcentral (my connection is working fine). I got valid hop info right up to and including the RC site itself (i.e no Time outs). I know that places do block ICMP packets but maybe RC isn't doing it uniformly?

It depends on what you're tracerouting. Ther is a load balancer in front of the webserver cluster. If you try going to a specific machine within the cluster, you'll get an ICMP timeout.

[nickb]

My tracert was directly to 'www.reefcentral.com':

tracert www.reefcentral.com

Tracing route to reefcentral.com [198.92.98.77]
over a maximum of 30 hops:

1 63 ms 16 ms 31 ms tlgw22.slnt.phub.net.cable.rogers.com [24.157.152.1]
2 16 ms 31 ms 16 ms 10.1.65.1
3 16 ms 31 ms 16 ms gw01-vlan966.slnt.phub.net.cable.rogers.com [66.185.93.213]
4 16 ms 31 ms 16 ms gw01.rchrd.phub.net.cable.rogers.com [66.185.82.133]
5 47 ms 15 ms 32 ms gw01.flfrd.phub.net.cable.rogers.com [66.185.82.129]
6 32 ms 31 ms 16 ms gw02.mtnk.phub.net.cable.rogers.com [66.185.82.125]
7 15 ms 31 ms 32 ms gw02.wlfdle.phub.net.cable.rogers.com [66.185.80.153]
8 47 ms 47 ms 47 ms dcr1-so-3-1-0.NewYork.savvis.net [206.24.207.85]
9 31 ms 63 ms 47 ms agr3-so-4-0-0.NewYork.savvis.net [206.24.207.74]
10 31 ms 47 ms 63 ms acr2-loopback.NewYork.savvis.net [206.24.194.62]
11 47 ms 47 ms 47 ms so-6-3.core2.NewYork1.Level3.net [209.244.160.189]
12 31 ms 47 ms 63 ms ae-0-51.bbr1.NewYork1.Level3.net [64.159.17.1]
13 78 ms 78 ms 94 ms so-2-0-0.mp1.Tampa1.Level3.net [209.247.11.201]
14 78 ms 78 ms 94 ms unknown.Level3.net [64.159.1.142]
15 78 ms 78 ms 78 ms s0-1.rs3.bbnplanet.net [4.24.183.138]
16 78 ms 94 ms 78 ms tito.rapidsys.com [209.84.253.239]
17 78 ms 79 ms 78 ms 198.92.98.77

Trace complete.

[MalHavoc]

www.reefcentral.com is the load balancer.

[nickb]

OK. But, if you look at the first Tracert (On page 1 of this thread), that was going to the same machine that my tracert went too. They got a 'time out' while I didn't. Hence, it doesn't seem right to explain their time out as due to the RC load balancing and ICMP blocking. That's my only point.

[MalHavoc]

If you're talking about Chad's post, that timeout is occuring at a different router. There is a .bbnplanet.com router right after the tampa*.level3.net routers. Then one of two possible .rapidsys.net routers (either rsrouter or tito). Then the load balancer (the .77 ip). Chad's timeout was occurring on the unknown tampa router.

8 96 ms 83 ms 96 ms ge-6-1.hsa1.Tampa1.Level3.net [64.159.1.14]
9 * * * Request timed out.

That was probably a different issue altogether.

I set up the machines behind the load balancer, so I know they are dropping ICMP indiscriminantly.

[Samw]

RC is now load balanced? This is new right? Is performance of the website much improved?

I think nickb's point is that no one here is tracerouting to a specific machine. Everyone is tracerouting to the loadbalancer and some of the guys on Shaw can and some can't traceroute to the loadbalancer. Therefore, the fact that the actual servers drop ICMP probably doesn't explain what is being reported here.

[nickb]

I doubt that this is worth pursuing but, your intial post said:

'As for the hop timing out after the Level3 network - that's normal. We block ICMP packets going into the RC servers so traceroutes appear to time out after that point. '

My point was that the time out occured BEFORE your ICMP dropping starts. If you compare my tracert info, there are three more hops before reaching the RC load sharing computer. Hence, the time out on the original tracert likely gives some indicatation of where the system failure is occuring.

[MalHavoc]

Originally Posted by Samw

RC is now load balanced? This is new right? Is performance of the website much improved?

I think nickb's point is that no one here is tracerouting to a specific machine. Everyone is tracerouting to the loadbalancer and some of the guys on Shaw can and some can't traceroute to the loadbalancer. Therefore, the fact that the actual servers drop ICMP probably doesn't explain what is being reported here.

I realize. We know what the problem is (I posted it in my initial response). We've always dropped ICMP at the server level. The problem is with a router going into rapidsys itself. Not on the way in, actually, but on the way back out.

[MalHavoc]

Originally Posted by nickb

I doubt that this is worth pursuing but, your intial post said:

'As for the hop timing out after the Level3 network - that's normal. We block ICMP packets going into the RC servers so traceroutes appear to time out after that point. '

My point was that the time out occured BEFORE your ICMP dropping starts. If you compare my tracert info, there are three more hops before reaching the RC load sharing computer. Hence, the time out on the original tracert likely gives some indicatation of where the system failure is occuring.

As I said in my initial post, the problem is with a particular router going into the ISP. It has nothing to do with our servers, or with ICMP. The fact remains, though, that those reports of dropped routes before you get into the various .rapidsys.com aren't what is causing this particular issue. This issue is with a specific bridge going into the ISP. It doesn't even have an IP address, so it doesn't show up on a traceroute anyway.

I managed to get a hold of someone at the ISP (in Florida). He's ticked that he's in there on the fourth of July, but oh well. Hopefully it'll be fixed today or tomorrow.

[Samw]

Originally Posted by MalHavoc

I realize. We know what the problem is (I posted it in my initial response). We've always dropped ICMP at the server level. The problem is with a router going into rapidsys itself. Not on the way in, actually, but on the way back out.

That's fine. I think we understand that you have routing issues. It was just confusing when you brought up the issue about dropping ICMP at the server level when no one had pinged or tracerouted to them. I'm glad to hear that RC is now loadbalanced. It was something I wanted to see them do to improve performance rather than just buying 1 big server. I hope it is working out. I haven't used RC in months so I don't know how things are going.

I run a small network myself with dial-up modems as part of the network. When I lost my main HIGH SPEED route for a week, I had to reroute everything through a 33.6k connection and run IP Masquerade instead of Proxy Arp for my modem users. That was fun seeing multiple modem users, the webserver, the DNS server, and the mail server, all sharing a low speed 33.6kbps route.