today the network at our LA office network took a giant crap. ports were going up and down on the switches, packet loss was incredible, latency was high, overall things sucked. as is their nature, our 3com 5500g’s didn’t say much other than that there were a few mac collisions on this or that port.
i started out doing what i typically do when there’s a network outage (other than freak out): i logged into our trusty nms (jffnms) to look at the traffic stats for all of the switch ports. almost all of the graphs for all of the ports on our switches resembled this one:
LOTS of output traffic. something had to be sending all of this traffic. i started scrolling down the graphs to look for which port was transmitting all of this traffic:
holy crap, what the hell is that? i shut down the port and had someone trace the cable back to the patch panel (which, by the way, is a pain in the ass. it’s really important that you keep your port:port mappings consistent).
we tracked down the port to an empty office (by empty i mean a desk and a voip phone). the voip phone was plugged into both network ports in the office… our voip phones (which are altigen phones, which suck, if you’re in the market for voip equiptment, do NOT go with altigen) have two ethernet ports, one to go into the wall (network) the other to go into a computer, thus acting like a network bridge. this is supposed to save you from having to add a network jack for each phone as you can just chain the phone in line with the computer’s network connection.
what we had was:
which was creating this traffic pattern:
normally spanning tree (STP) would keep this from grinding our network to a halt… but it didn’t. either because we don’t have stp enabled on our switches, or because these ports are not enabled as edge ports, or, the phone was stripping the stp packets.
