CPS was recommended to a local division of MCI/WorldCom (hereafter referred to as WorldCom) as an expert in Wide Area Networks and Security. A meeting with David Caldwell would result in an interesting and complex project involving six (6) of their Fortune 50 clients IVR systems implemented and managed by WorldCom.
Contractual Obligations
Each of these clients relied on WorldCom to provide IVR systems with redundant sites located throughout the eastern half of the United States. Client's requirements follow:
- Each client's IVR systems were to be securely separated.
- All IVR's to be centrally managed within WorldCom's GA. data center.
- All IVR's and networks to be centrally monitored by WorldCom's management operations center.
- Redundant sites to ensure 24x7 access for each customer's IVR systems.
Problems
- The solution designed and implemented by a WorldCom employee that who was no longer there, leaving no documentation.
- The Wide Area Network utilized a combination of routing protocols: RIP, RIP v2, IGRP, OSPF and BGP. These protocols implemented in a haphazard and seemingly senseless manner.
- Route fluttering was occurring every 20-30 minutes, with loss of some routes for as long as 4 hours.
- HSRP was implemented for router redundancy at the GA. data center, but was incorrectly configured. Ultimately, this contributed to some of the longer network outages.
- Security between customer sites was designed around a Check Point firewall, yet routers were implemented in such a manner as to totally by-pass the firewall.
Challenges
- No documentation.
- Map the network and obtain configurations for all routers, while simultaneously dealing with consistent route fluttering.
- Make all changes from the GA. data center, no travel budget to remote data centers.
CPS Solution
- Once the collation of configurations was accomplished, manually draw a picture of the network and routing scheme.
- Upon developing a solid understanding the total environment t. CPS decided to reduce the routing protocols to two: EIGRP and BGP.
- Disable HSRP temporarily at the core routers.
- Working with one edge router at a time, and the core router, stabilize the routing environment through the use of static routing.
- Re-configure the firewall to maintain integrity between customer networks.
- Re-write each configuration file for each router.
- Upload new configurations with new routing protocols at the core (while maintaining static routing), and implementing at one edge router at a time.
- Remove static routes from the edge to the core.
- Implement least cost routing for redundancy.
- Implement HSRP properly at the core routers.
- Finally, Redistribute EIGRP into the BGP cloud, integrating WorldCom's monitoring network.