routing issue for verizon dsl customers in western massachusetts

[morrowc.lists at gmail.com (Christopher Morrow)]

On Thu, Sep 15, 2011 at 4:13 PM, Steve Bohrer <skbohrer at simons-rock.edu> wrote:
> On Sep 15, 2011, at 3:39 PM, Christopher Morrow wrote:
>
>> On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold <bgold at simons-rock.edu> wrote:
>>>
>>> Over the past week, we've discovered that there is an issue with the way
>>> some Verizon DSL customers are being routed in Western Massachusetts that
>>> is
>>> preventing them from reaching my employers public IPs. The problem is
>>> only
>>> limited to Verizon DSL customers, everyone else can reach these IP
>>> addresses
>>> just fine. After many hours on the phone with Verizon tech support, I
>>> finally managed to get myself and one of my coworker's home dsl
>>> connections
>>> switched from a "redback router" to a "juniper router" which resolved the
>>> issue, but only for us.
>
> [...]
>>
>> If you buy verizon services at your day job you can probably make
>> noise through your sales droids better than here (sadly)... verizon
>> likes to jump when customers have problems, if the customer is a large
>> corporation or other 'important' customer.
>
>
> That is just the problem! The college does not buy any Verizon network stuff
> directly, so we don't really have any access to their support. (We have a
> few cell phones, but not enough to be "important".)
>
> Brian Gold (who first posted) happens to have their DSL to his house, and he
> was one of five who have reported the problem, so that gave him a slight in.
> But the only techs he could reach as an "end user" were not high enough up
> to fix this problem in a general way. After pressing them for literally
> hours, he was able to get transfered to their NOC, and get the problem
> resolved for his one address. But, they would not give him the NOC contact,
> and he had to repeat this multi-hour process to get it fixed for an other
> user. Verizon's DSL support suggested that we get our bandwith provider
> involved, and so they tried to pitch in, but they don't have any Verizon NOC
> contact either, especially since this issue is purely within a small corner
> of Verizon's DSL network, not on any of Verizon's links to our provider.
>
> This issue hits only a few Verizon DSL users in NW Mass. It does not really
> seem like a routing problem, because the affected users can reach many of
> the servers in our AS, but not some addresses. Unfortunately, the "blocked"
> addresses include our web server and our mail server, so our staff who live
> out there noticed the issue pretty quickly. Traceroutes from Brian's house
> show that for our blocked hosts, the users don't get beyond Verizon's NAT.

I wasn't aware verizon implemented CGN already... way to be a 'first
mover' in this field verizon!

>
> The Verizon tech's "fix" of re-patching Brian's DSL line in to a different
> router feels to me like there is a config problem in the other router, but
> the tech we got is not authorized to alter the config. It would be nice if
> we could reach someone who could actually edit the broken config and make it
> right. Anyone from Verzion's NOC for Western Mass reading this? Or, does
> anyone else have useful contact info for them?

you probably want someone in the NOC which is (I think) stil in
Reston, va... I don't think they have separate noc's per region. The
first-line tech folks you chat with on the phone really arent' able
(even to login really) to fix devices in the field :(

anyways, this looks crappy :( but yeah for CGN being all it's cracked up to be!

-chris

>
> FWIW, Simon's Rock is 208.81.88.0/21, AS 19345. Here are a failed and a good
> trace from Brian's house, to different servers on our campus :
>
> FAILS:
> Tracing route to wilbur.simons-rock.edu [208.81.88.15]
> over a maximum of 30 hops:
>
> ?1 ? ?<1 ms ? ?<1 ms ? ?<1 ms ?192.168.10.1
> ?2 ? ? 1 ms ? ? 1 ms ? ? 1 ms ?192.168.1.1
> ?3 ? ?53 ms ? 104 ms ? 116 ms ?10.14.1.1
> ?4 ? ? * ? ? ? ?* ? ? ? ?* ? ? Request timed out.
> ?5 ? ? * ? ? ? ?* ? ? ? ?* ? ? Request timed out.
> ?6 ? ? * ? ? ? ?* ? ? ? ?* ? ? Request timed out.
> ?7 ? ? * ? ? ? ?* ? ? ? ?* ? ? Request timed out.
>
> WORKS:
> Tracing route to dev.simons-rock.edu [208.81.88.25]
> over a maximum of 30 hops:
>
> 1 ? ?<1 ms ? ?<1 ms ? ?<1 ms ?192.168.10.1
> 2 ? ? 1 ms ? ? 1 ms ? ? 1 ms ?192.168.1.1
> 3 ? ?87 ms ? ?54 ms ? ?54 ms ?10.14.1.1
> 4 ? ?99 ms ? 109 ms ? 103 ms at-0-3-0-1711.WMA-CORE-RTR2.verizon-gni.net
> [130.81.10.77]
> 5 ? ?16 ms ? ?18 ms ? ?16 ms ?so-7-3-1-0.NY5030-BB-RTR2.verizon-gni.net
> [130.81.20.6]
> 6 ? ?19 ms ? ?17 ms ? ?17 ms ?0.xe-3-1-0.BR3.NYC4.ALTER.NET [152.63.2.81]
> 7 ? ?18 ms ? ?21 ms ? ?18 ms ?204.255.168.194
> 8 ? 108 ms ? 188 ms ? 116 ms ?pos5-0-2488M.cr1.BOS1.gblx.net [67.17.94.57]
> 9 ? ?24 ms ? ?28 ms ? ?23 ms ?pos0-0-0-155M.ar1.BOS1.gblx.net [67.17.70.162]
> 10 ? 121 ms ? 160 ms ? 127 ms ?64.213.79.250
> 11 ? ?77 ms ? ?77 ms ? ?78 ms ?208.81.88.25
>
> Trace complete.
>
> Anyways, thanks for any suggestions you can offer.
>
> Steve Bohrer
> Network Administrator
> ITS, Bard College at Simon's Rock
> 413-528-7645
>
>
>
>