Service Delivery
Experience

Telefonica – Three Continent Global Networking

It’s 2001 and global demand for the World Wide Web (Internet) and private data networking is exploding. Telefonica S.A., the Spanish Global 500 telecommunications giant, has just completed a 5-year acquisition of the majority of local dial tone in South America. Already one of the largest data network providers in Europe, and now the largest data network provider in South America, Telefonica needs to add North America to the list of continents served by its global data network.

Two years earlier, in 1999, Telefonica formed a joint venture with Tyco International to build a $900M, 23,000-kilometer, state-of-the-art undersea Dense Wavelength Division Multiplexing (DWDM) fiber optic network. The network begins in Miami, FL, traverses the east coast of South America, crosses the Andes to the west coast, along which it runs until it is brought through Guatemala, the Caribbean, and finally back to Miami. The purpose of this network is to connect the major population centers of South America together to provide high-speed Internet and private data networking services to multi-national corporations with operations in South America, North America, and Europe.

At the time of the construction of this fiber optic network (press release), Telefonica did not have a U.S. data center from which its multi-national corporate customers would be able to connect to its South American, European, and U.S. data networks. When construction on the South American fiber optic network began, Telefonica also started building its Tier 1 U.S. “KeyCenter” data center in Miami, FL. Miami was selected because the South American fiber optic network originates and returns a few miles north of Miami in Boca Raton, FL.

The South American fiber optic network was completed in March 2001 (press release). However, due to delays in site selection and architectural design, the KeyCenter had just begun construction when the South American fiber optic network was completed. During this time Heydrick & Struggles had been hired by Telefonica to recruit an executive team to startup operations for a new Telefonica subsidiary, Telefonica Data USA. This new U.S.-based subsidiary would operate and manage Telefonica’s U.S. and Puerto Rico data centers and sell data services delivered from these data centers to multi-national corporate clients located in both North and South America.

I was part of the 4-person executive team that was recruited by Heydrick & Struggles. In my position as Vice President – Data Center Facilities I was responsible for the construction and operation of all U.S data centers. My first assignment was to complete the construction of the Miami KeyCenter. Once the KeyCenter had been built and was operationally stable I was responsible for building similar data centers in the largest Hispanic population centers in the U.S. This included New York City, Houston, Los Angeles, Chicago, and Dallas.

Miami KeyCenter construction began in early March 2001. Because the South American fiber optic network had already been completed, and was awaiting the completion of the KeyCenter data center to be placed into service, my first assignment was managing the completion of KeyCenter construction. I oversaw construction that ran 24 x 7 until the KeyCenter was completed in late May 2001 (90 days). I was also responsible for construction financing including approval of all general contractor invoices against the written design specification produced by our architecture firm.

When it was completed, the Miami KeyCenter occupied 22,000 square feet of Tier 1 data center space within a 176,000 square foot tilt-up concrete building that was certified to withstand Category 5 hurricane winds. The building was located 23 miles inland from the Atlantic Ocean to avoid hurricane tidal surge. Everything was fully automated and redundant. Building controls were computerized by Johnson Controls. Security personnel were present 24 x 7 and operated within a bulletproof glass enclosure. KeyCenter access on all doors, including interior doors, was controlled by keycard access that logged the identity of every person who passed though any KeyCenter door. An on-site 24 x 7 Network Operations Center (NOC) monitored client network health as well as application health. The NOC had visibility to all of Telefonica’s global networks, including South America and Europe, to ensure that Service Level Agrrements (SLAs) could be met.

After construction of the KeyCenter had been completed I worked with my facilities team to self-commission the KeyCenter, saving Telefonica $500,000. This process involved testing and certifying that all systems had been correctly installed and operated in accordance with the design specifications produced by the KeyCenter architect. This process took just over two months to complete and was 100% successful. The KeyCenter Grand Opening was held on September 19, 2001, just 8 days after the 911 terrorist attack on the World Trade Center in New York City. Governor Jeb Bush, Miami Mayor Alex Penelas, and other South Florida and Telefonica dignitaries attended the event.

In addition to functioning in my data center facilities role I also acted as the product manager for KeyCenter service offerings whose written documentation I designed, wrote, and published. The primary service sold during the initial months of KeyCenter operation was Dedicated Collocation Service. Our goal was to sell blocks of floor space that were dedicated to large multi-national corporations who would use it to power their Internet and private network equpiment (i.e., “pipe, ping, and power”). These clients would also purchase Internet and private data network connectivity from us in order to reach their operations located in the U.S., South America, and Europe.

The results of the executive leadership team hired by Telefonica to launch Telefonica Data USA were impressive. We sold and delivered $35M of data services during the first year of KeyCenter operation. The Total Contract Value of these services was $110M. Telefonica Data USA was EBITDA-positive in the 7th quarter of its operation. The KeyCenter uptime during the entire time I was responsible for its operation was 99.99999%.

Celanese Chemicals – Global IT Infrastructure Replacement

It’s June 1997 and Celanese Chemicals has become a global manufacturer and supplier of commodity chemicals. With 30 manufacturing sites and 12,000 employees located in seven countries, the company is a $5 billion enterprise that supplies the basic ingredients for everything from toothpaste and plywood to lipstick and ibuprofen. Orders are being received from around the world but the company’s IT infrastructure was built when it was a regional manufacturer that served only the European and North American markets. It now realizes that it needs to redesign its entire IT infrastructure to support its new global business model.

To solve this problem Celanese Chemicals launches its GlobaLink project. The purpose of this project is to replace its existing global IT infrastructure with state-of-the-art technology that will enable the enterprise to conduct business between any of its 30 sites located around the globe. To accomplish this goal, Celanese Chemicals realizes that it must standardize its technology infrastructure so that it can provide reliable 24 x 7 global application connectivity. To this end Celanese Chemicals decides to utilize a suite of Microsoft products that will deliver a standardized, globally networked application infrastructure.

Because each Celanese Chemicals site management team has decision-making responsibility for their local IT infrastructure, the GlobaLink project contains a required component and an optional component for each site. The required component is that at least one Windows NT Server has to be installed at each site. This server will provide the Active Directory (AD) functionality needed by the global Celanese Chemicals community to enable site-to-site application connectivity. The optional component of the project is that local site management teams can purchase a Celanese-approved, Microsoft-based infrastructure solution (more on this later) or deploy another technology of their own choosing that can interoperate with the GlobaLink network.

In addition to the global network component of the GlobaLink project, Celanese Chemicals had made the decision that it would upgrade the IT infrastructure of its U.S. headquarters using a comprehensive suite of Microsoft technologies and applications. This meant that 400 users’ desktop PCs and laptops would be upgraded to Windows NT Professional and Office 97. Windows NT Servers, Exchange Server, Systems Management Server, and SQL Server would provide the technology upon which Celanese Chemicals would deploy both its SAP R/3 supply chain solution and its office productivity applications. This included global e-mail provided via the Outlook e-mail application included with Office 97.

Celanese Chemicals engaged Microsoft Consulting to validate the design and long-term supportability of their GlobaLink technology solution. What Microsoft could not do was assist Celanese Chemicals with the deployment and ongoing support of its GlobaLink solution. For that role Celanese Chemicals tapped Vanstar Corporation, a large-scale PC reseller and systems integrator who had been providing PC break-fix services to their U.S. headquarters for two years prior to the start of the GlobaLink project. Vanstar was also a Microsoft enterprise alliance partner, which meant that Vanstar and Microsoft already had a strategic working relationship well before the GlobaLink project was conceived. This provided Celanese Chemicals with the reassurance of knowing that the two most significant partners in their GlobaLink project had already established a business relationship that was working well and whose focus was the success of their shared customers.

Vanstar and Microsoft collaborated on the GlobaLink technical design. Vanstar created the design and Microsoft provided technical input and validation of the Vanstar design. Vanstar used its HORIZON solution development methodology to guide the creation the GlobaLink technical design plus the implementation and post-implementation tasks and processes that would be needed to deliver a successful GlobaLink project. Use of the HORIZON solution design methodology ensured that the Celanese Chemicals technical and business visions for the GlobaLink solution became a reality upon completion of implementation. I was the Vanstar Solution Architect responsible for developing the GlobaLink technical solution, creating and managing its global implementation, and creating its post-implementation support processes and systems. I functioned in a dual role as both the Solution Architect and Project Manager during all phases of the GlobaLink project.

Prior to the GlobaLink project I had become one of only three solution design methodology subject matter experts at Vanstar. This team had developed Vanstar’s proprietary HORIZON solution design methodology that was used throughout Vanstar’s Professional Services Organization (PSO) to create highly complex technical solutions for Fortune 2,000 clients. HORIZON solutions contained implementation and ongoing support components because, in addition to professional services, Vanstar sold hardware and software, and ongoing break-fix support services. This meant that the HORIZON methodology would be an ideal tool with which to design a project like GlobaLink that required technical design, hardware and software, implementation, and ongoing support. Vanstar supplied all of these components to the GlobaLink project.

I created a Microsoft Project project plan for the GlobaLink project using a Work Breakdown Structure (WBS) that was modeled after the Vanstar HORIZON solution development methodology. Because the GlobaLink project was complex, I knew that the best approach to use to define the scope of work for the project, and therefore its cost, was a collaborative discussion between Vanstar and Celanese Chemicals. I created the GlobaLink project plan using resource-based task definitions and added the “Cost” column to the plan. This enabled both Vanstar and Celanese Chemicals to understand the Work Effort and Cost for each of the 495 tasks contained in the final version of the GlobaLink project plan. I printed the project plan on three 36″ x 24″ plotter sheets and, working with the Vanstar sales team, used the plotter sheets to negotiate the final scope of work and price for the project. Celanese Chemicals IT executives were partners in establishing the final scope of work. That meant that if the project experienced problems during implementation that both Vanstar and Celanese Chemicals had a shared interest in immediately correcting those problems.

Once the project plan had been developed and approved by all parties to the GlobaLink project, the first major project deliverable was the technical design of the GlobaLink network. After the design was completed by Vanstar, and approved by Microsoft and Celanese Chemicals, Vanstar worked closely with Celanese Chemicals to define and build a GlobaLink test lab. The purpose of this lab was to test the various hardware and software configurations that would be deployed to the 30 Celanese Chemicals sites worldwide plus its Dallas, TX headquarters building.

While the test lab was being defined and built, Vanstar worked with Celanese Chemicals to develop its universal Windows NT site specification. This specification established the minimum technical requirements that each of the 30 global Celanese Chemical sites must meet when they installed their Windows NT GlobaLink server. It also provided a step-by-step guide for defining a site-wide Microsoft computing infrastructure, should they desire to use Microsoft technology as the platform for their computing infrastructure. This specification was documented in the Site Implementation Kit document which I designed, wrote, and produced.

The GlobaLink implementation began in September 1997. Each of the 30 remote sites installed their Windows NT Server, which provided Active Directory services for their site to the greater GlobaLink network. They also either upgraded to a Microsoft site computing platform (using the Site Implementation Kit document) or interfaced their existing Novell systems with their Windows NT Server. At the same time, the upgrade of 400 users at the Celanese Chemicals Dallas, TX headquarters building began.

A trial had already been run that tested the effectiveness and efficiency of the end user Windows NT upgrade process during which the kinks had been worked out of the process. When the Dallas headquarters office building end user upgrade project began it was executed with speed and accuracy. The primary goal of the upgrade process, aside from upgrading each end user’s technology, was to deliver a fully functioning PC that met each person’s individual needs. This included custom applications that were unique to each end user’s work situation.

End users were scheduled to attend a 4-hour afternoon Windows NT training class. The class instructed users how to use their new Windows NT operating system as well as their new Office 97 applications. Before they attended class they delivered their laptops to the PC upgrade area. This was a two-conference room whose divider had been retracted. End users who were using a desktop PC had their PC retrieved by a Vanstar PC technician while the end user was attending training class. When the end user returned to work the following morning their newly-upgraded PC was installed at their desk and ready for them to use. This approach meant zero PC downtime for end users.

The upgrade process started by separating PCs that would be upgraded from those that would be decommissioned and replaced by new hardware. Hard disk drives were wiped clean so that no data could be retrieved (end users had been responsible for backing up their data to the Celanese Chemicals servers before their PCs were upgraded). Each PC went through the same rigorous upgrade process. Software images that had been developed in the test lab were installed onto laptops and desktop PCs. Each end user had previously been associated with a specific software image so the Vanstar upgrade team knew exactly which software image needed to be installed on each end user’s PC.

A comprehensive test was performed on every upgraded PC, including testing to ensure that each end user could launch any custom applications that were required for them to perform their job function. Only when an upgraded PC passed its testing was it certified as upgraded and returned to its end user. A small tri-fiold brochure that I had previously developed for the Celanese Chemicals IT department (during a professional services engagement to improve the quality of their PC support service) was left with each PC so that the end user knew where to call to get support if it was needed. A staff of Vanstar PC support technicians provided “day after” support services to end users. Because of the rigorous and comprehensive upgrade and testing processes the number of “day after” calls were minimal.

The results of the GlobaLink project were noteworthy. Redundant applications in the Celanese Chemicals software environment were reduced by 50%, which was directly attributed to reducing IT end user support costs by 10%. Celanese Chemicals was so pleased with Vanstar’s performance that it contracted with Vanstar to provide 100% of its end user PC support, estimating that the contract would save its IT department 30% over what it was spending to provide this support with its own staff. For Vanstar, the GlobaLink project led me to create Vanstar’s Advanced Desktop Renewal System (ADRS), a stadardized PC refresh methodology that was adopted Vanstar-wide and became the standard by which Vanstar delivered all PC refresh projects. Microsoft wrote a Solution Overiew case study that was published on both the Microsoft and Celanese Chemicals Web sites. I was quoted extensively in this case study (my quotes are highlighted in yellow).