A snafu during a planned upgrade at MAE-West, one of the nation's key switching facilities for Internet connectivity, led to a serious interruption of service early Sunday morning.
The outage caused slow - or, in some cases, no - Internet throughput for many ISPs until early this afternoon, when engineers finally were able to harness the situation.
The official explanation was that an unexpected problem with a needed upgrade to the facility caused the temporary outage.
"The ISP customers need more bandwidth, and the gigabit switches were at their maximum limit as far as bandwidth," said Linda Laughlin, spokeswoman for WorldCom, the telecommunications giant who owns the San Jose facility. "With the upgrade over the weekend, there were a couple unexpected delays for customers."
MAE-West is one of two major network traffic exchange points in the United States. The other, MAE-East, is in Vienna, Virginia. WorldCom owns and operates both these and several smaller-scale Metropolitan Area Ethernet facilities throughout the country; ISPs connect routers to these MAEs to exchange data at very high speeds with other networks.
These MAEs are major Internet traffic junction points - as well as centers of vulnerability. Almost a year ago, a programming blunder on a MAE-connected ISP's router caused a major Net outage when it propagated out to MAE-East.
As part of a plan to expand MAE-West, the entire facility was physically moved down one floor in the building where it was housed. Laughlin said that technicians began prepping late Friday night for the move to the 10th floor, with the actual changeover taking place late Saturday night and very early Sunday morning.
"We tried to pick off-peak hours, and there actually aren't many off-peak hours in the Internet world," she said.
It was during this transition that an unexpected equipment problem brought MAE-West's operations down, with network traffic forced to route around it - making for slow connections all around.
"Instead of going across that peering point, we have to ship all that information back across MAE-East, said Jason Clement, a network engineer at GoodNet, one of the many affected ISPs.
It was so bad that a technician at GridNet, a Internet service wholesaler and subsidiary of Worldcom, said that six straight minutes of connectivity was the best he had seen all day, while a technician at Best Internet's network operations center said their users have experienced slow connections - or none at all.
"Some packets routed around MAE-West, others went into it and died," he said. "The people I really feel sorry for are those who only have connectivity through MAE-West."
Things finally began to stabilize late this afternoon.
"I just talked to the engineers and they said that there might be some degradation right now, but all customers are back up," said Laughlin. "There might be a few delays right now as they bring everything else up.
Laughlin said that WorldCom is building a new, ATM-only MAE in Dallas, MAE-Central, which will also be one of the major MAEs. Service should be available some time later this year.
