Data center relocation project

Early in 1992, Rockwell International elected to close its Texas computing center and consolidate its function into the computing center at Rockwell's corporate headquarters in California. The resulting project would take more than a year and would involve transferring computing hardware and software libraries, employee downsizing, and computer reconfiguration—all while providing continuing service to Rockwell's customers and divisions. Rockwell accomplished its goals in a classic example of project management.

Announcement

On February 15, 1992, the decision to relocate Rockwell's Southwestern Computing Center (SWCC) to California was announced by the vice president of Rockwell's Information Systems Center (ISC). The actual transfer of work was to take place in two stages, over Thanksgiving and over Christmas, to minimize customer impact. The project manager who was to close the Texas center also headed the center.

The announcement acknowledged that the relocation was “a complex project with some risk of business interruption,” requiring “extreme attention to detail in order to minimize any negative impact.”

Project personnel would be drawn from ISC's functional staff; the project director's authority would be strengthened by reporting directly to the vice president of operations.

From the outset, the project's objective was clearly defined. An announcement was released specifically outlining what was to be accomplished.

Project Start-Up

March-April 1992. During start-up, project offices were established in Texas and California. The technical team was drawn from both sites. Other divisions were notified that their technical experts would be called to participate in “Red Team” reviews. At the same time, plans were negotiated with outside vendors to facilitate staging the move.

Major studies were in full swing, ranging from how to provide print and job control services to customers while the move was in progress to redesigning the new data network. In March, test dates were already being scheduled for systems testing during the first workload transfer in November.

Figure 1. Overview of Organization for SWCC Relocation Project

By the end of April 1992, planning was 99 percent complete. The first Red Team review was scheduled for June. The appropriation request had been approved and was being routed for signature. Staff at both sites were being cross-trained, and workload forecasts had been completed. The people in Texas already knew which of them would be laid off and when.

Work Breakdown Structure. The project was broken into three primary areas. The first was consolidation of the technical support/function; the second was transitional operations; the third, and most difficult of all, was workload transfer with all its attendant problems of hardware, facilities, systems, network, vendor support, finance, customer support, service transfer, operations, and quality assurance (see Figure 1).

Each of these three primary sections was then dealt with in a work breakdown structure. Each was assigned appropriate personnel within each functional department.

By April 1992, each section had been defined as a unique work package, with its own manager and itemized breakdown.

At the same time milestones were being defined and set on the calendar, taking the project from initiation on March 1, 1992, through the transfer of work in November and December, to final closing of the Texas site in 1993 (Figure 2).

The Project

May-August 1992. Throughout the summer, studies were undergoing review. As problems were identified, alternate options were developed and considered. Meetings with customers continued. Initial staging was already under way.

The Data Network Department had determined optimal rerouting of data circuits and T-1 digital communications links at Texas and began initial staging in June. Because the rerouting would impact Rockwell's Allen-Bradley division, management conducted customer meetings; then staging and test dates were set to meet Allen-Bradley's concerns.

To control Rockwell's entire SNA network, the Communications Management Configuration (CMC) computers had been configured to operate concurrently. These now had to be collapsed into a single system.

Alternate site processing strategies were studied. Sites were selected for emergency purposes in case of facility or electrical outages.

Some stages of the project functioned as separate units. For instance, Rockwell's tape library in Texas had its own timetable. Not only was it responsible for shipping thousands of tapes from Texas to California, it had determined that this was the optimal time to shift over to a specialized automated system to service Rockwell's libraries.

Throughout this period, while the studies were being reviewed and refined, the development of items in the work package were ongoing. Every item listed in the work breakdown structure was now identified, defined, and assigned to a manager; a schedule of completion dates was also in place. Each week a “Weekly Work Package Status” bulletin was published, updating each upcoming item scheduled for completion within the next two weeks. Each item was listed by its WBS number. The milestone was identified, as was the manager responsible and the date it was projected for completion (see Table 1).

Figure 2. Major Milestones for SWCC Relocation Project

September 1992. The project was accelerating and the pressure was on. The first major workload transfer was only two months away. On September 1, mainframe configurations were finalized. By September 8, the transfer schedule was published, including dates set aside where no transfer could take place because of Space Shuttle launches.

The tape library was issuing move schedules. Tapes were identified by shipment and arrival dates, whether they would be coming by truck or air, the weight of each box, and the cost of each shipment.

At the same time, the hardware department was issuing its own timeline for shipping its computers.

As the project moved into planning full-blown execution, meetings were now held weekly, linking California and Texas in videoconference. Managers were encouraged to voice their concerns. Fallback plans were in place, as were dates for systems testing for the first major move.

Table 1. Sample Page of Weekly Work Package Status

Table 2. Sample Page of Rockwell's Information Systems Center User News

By mid-September, major milestones had been issued for all principal areas. The project was on target.

Meanwhile, key Texas personnel were already accepting job offers with other companies and the Texas support base was rapidly dwindling. It would not be easy to make the workload transfer without them, so contingency staffing plans were set up.

The California tape library prepared for the first major shipment of tapes in October. Modifications were ongoing. Small truckloads of shelving and staging equipment were now arriving, and surplus PCs from Texas were being relocated. Departments began to fine-tune weekly status reports down to the supervisor and staff member level, specifically naming work package items, projecting completion dates, and identifying the staff members responsible.

On September 22 one of a series of “All Hands Meetings” was held. The purpose was to inform everyone of the next major phase of the project—staging the November transfer. Timelines were issued showing how the Hardware and Systems departments would work together to install and test equipment. Configuration plans for computer network pathing were explained to all groups involved in the process. As one center would shut down, plans were in place for another to be brought up.

No project proceeds without a few problems, however: the plan to automate the tape libraries was running into difficulties. On September 25 it was discovered that some of the datasets scheduled for compression onto a more advanced system could be damaged by the transfer. Hurriedly, it was decided to copy them onto similar tapes temporarily and deal with actually transferring them after the relocation.

By September's end, everything appeared ready. The Red Team had reviewed the plans in June, now a second team was scheduled to make the final review.

October 1992. In its “User News” of October 1, Rockwell issued its formal announcement of the coming workload transfers for Thanksgiving and Christmas. Each transfer would require a 96-hour service outage. Computing addresses would be affected. Some system update cycles would be skipped (Table 2).

By mid-October truckloads of equipment were arriving. The 96-hour plan was being rebuilt to reflect added system volumes that would be generated by moving during the transfer window. The tape library had fine-tuned its timetable; it could now accommodate the truck-loads and planeloads of tapes that would be arriving, particularly during the crucial 96-hour time frame.

The Facilities department began setting up additional cooling systems, adding racks to accommodate the added volumes. Computer TCP/IP and DECnet address changes were already being tested and scheduled and hardware was being installed. Payroll was making plans to have employees' holiday paychecks ready despite the 96-hour freeze.

Arrangements were made for selected corporate systems that interfaced with Rockwell divisions so that they might remain scheduled for production during the critical 96-hour period.

Employees were notified of timecard charges for these 96 hours. Customers were also updated, given new customer service support numbers and their sources of temporary emergency assistance during this critical period.

On October 30 an interim financial report was issued. While projected savings were almost exactly as predicted, the relocation project was $660,000 over budget.

November 1992. By November 3, the Red Team had concluded its review and the strain caused by months of work was showing. One manager expressed concern over the high stress levels that the project was causing on the technical staff: “All of the operating systems staff has become a shadow function to the project.” It appeared that at this stage of the relocation project, regular work was going by the wayside in order to meet the increasingly demanding project deadlines. “Many of the line functions are adding staff resources to accomplish project tasks,” said a Rockwell memo dated November 3.

“96 Hours,” Thanksgiving. By the time Thanksgiving arrived, a command center was set up to monitor the relocation of one of Texas' smaller data units over the holiday. This transfer would serve as a trial run for the major move that would take place over Christmas. The command center would be the focal point for information, and would be staffed around the clock.

The critical time arrived. Staff were broken into teams with instructions to contact their team captain in case of problems. The captain had the command center's central number, and command center staff would log each problem they received. Orders were to either contact another technician to resolve the matter or, if it could wait, issue an order to resolve the problem at a later date.

An extension was set up to give recorded updates on significant project milestones, and the recording was updated every two hours round the clock. Status updates or questions by divisions would be called in to the command center itself.

The official staffing plan covered all shifts on each of the four days to ensure adequate coverage. Gantt charts, sorted by task ID as well as by start time, showed what would be going on in each hour. Accountability charts by start time and by group were also prepared.

December 1992. The Thanksgiving transfer went well, and clean-up activities were rapidly concluded. By December 9, 1992, all but three of the 22 problems encountered during the transfer had been resolved.

The 96-hour plan for the major relocation at Christmas was now being refined to reflect last-minute changes. Employees from the California center were being flown to Texas to temporarily fill the posts of the rapidly dwindling Texas workforce. In California, staff members from other departments were working in the tape library in round-the-clock shifts to cope with the volume of incoming tapes.

As the work pace peaked, a crisis suddenly loomed: production test dates planned months in advance were suddenly unacceptable to a major customer. Working with the customer, testing times were renegotiated and a new schedule was put in place. A listing of major milestones was issued showing each major area in relation to the others.

By December 23, everything was ready, and managers were starting to look ahead, compiling lists of project participants who would be recognized and thanked.

“96 Hours,” Christmas. Christmas arrived. The same scheme that had worked so well over Thanksgiving was again set in place. The plan worked smoothly, with the command center coordinating all activities. The coordinating plans were published just as they had been during the earlier 96-hour period, but on a much larger scale. Information was circulated on each element:

• Staffing plan by employee name
• Staffing plan by center/start time
• 96-hour Gantt chart by task ID
• 96-hour accountability by start time
• 96-hour Gantt chart by start time
• 96-hour accountability by group.

Schedules were issued showing hour-by-hour (and sometimes minute-by-minute) how each task would be performed and by whom: employees, vendors, centers, coordinating groups, managers, and systems as each was brought up or down. The operating systems of two data centers gradually became consolidated into one.

During the move literally hundreds of circuits linking more than 50 different sites were re-homed to California. Hardware was installed, and services were transferred. By the end of the 96-hour window, the Texas center was gone.

Looking Back

In the turmoil of the massive 96-hour Christmas move, amid the hundreds of circuits relocated, the hundreds of staff members coordinating the systems being brought up and closing down, what was the worst complaint received? On the first day back from the holidays, a few customers complained that the systems were a trifle slow coming up that morning.

January-March 1993. The new California site was now operating. The Texas center was no longer active; nevertheless, there were still hundreds of non-critical circuits that had to be moved. Throughout January, on an hour-by-hour basis, these circuits were accounted for and relocated. The T-1 digital communications links that had once serviced Texas were now canceled or moved to other Rockwell sites.

Project close-out began in earnest. By January 29, excess equipment was sold off or shipped to other divisions. Booking these sales reduced the amount the project was over budget from $660,000 to $320,000.

There were a few last-minute crises such as damage to some trucked equipment, some of the tape converters were late in coming, and of the thousands of tapes that were shipped, 38 were missing. On the other hand, the scheduled circuit re-homes were well ahead of schedule.

Although meetings would continue for another month, the relocation project was essentially winding down. In February those who had participated in the relocation project were presented certificates of appreciation on mounted plaques. In March 1993 the project was officially over.

Why the Project Succeeded

Based on its record, the project was a success. There were many reasons it worked so well. First, the project had a strong matrix organization, drawing heavily from Rockwell's functional areas. Second, the company put an experienced project manager in charge. The project manager had already successfully managed the consolidation of two other centers in 1985, and was director of the Texas center that was to be closed down. The project manager's substantial authority at the Texas site was bolstered at the California location through active involvement from the advisory council. Furthermore, reporting directly to the vice president for operations at the Rockwell Information Systems Center helped in getting things done more smoothly.

Decentralized work sites also helped. Project offices were established in both Texas and California, allowing on-site control. People from both the Texas and California computing centers served on the technical team, enforcing from the start a spirit of cooperation.

A smooth work breakdown structure was another asset to the project. The project itself was broken into three primary areas. Each was further subdivided into workable units with clear delegation of responsibility. The project was also adequately staffed, and there was a fulltime project administrator assigned, reporting directly to the project manager.

Budgeting. Financial requirements for the relocation project were carefully planned from the outset. The budget was tracked and variances were regularly accounted for in management meetings. Estimates were accurate: the original estimates turned out to be very near the final figures. The estimated net savings of $49.87 million was re-forecast at $48.45 million by January 1993. With start-up costs of $3.92 million, the project was only 8 percent over budget.

Resource Management. A strong point in planning was resource estimating and management. Project resources were determined by first defining the project, then breaking it into component parts. Each part was in turn assigned to managers who broke the parts down further into their component tasks. Via the Weekly Work Package Status reports, management representatives were faced with continual deadlines ensuring task completion.

Top-down and bottom-up planning resulted in a realistic estimate. What this finally came down to was the detailed “96-hour” schedules that identified each task by its WBS ID number, by group responsible, by person responsible, and by estimated start and finish times (often minute-by-minute). This entailed the organization of hundreds of staff members from dozens of different departments and two widely dispersed geographic centers. In addition, thousands of circuits, lines, T-1 digital communications links, hardware, truckloads and planeloads of tapes, and computer equipment all had to be individually identified, planned, moved, tested, and verified. But the detailed planning efforts spent up-front paid off in smooth project execution.

Management of Conflicts. Conflicts are a normal part of projects. In this one, conflicts were handled in weekly management meetings, many of them were via videoconference between California and Texas. At times friction occurred. The operations staff especially experienced stress and their manager repeatedly voiced concern. Functional responsibilities were being short-changed because staff was absorbed by the ever-increasing project demands. In the face of approaching deadlines, however, staff from other departments were volunteered to help. During the massive influx of tapes into the California tape library especially, other departments were “commandeered” in four-hour shifts to shelve truckloads of tapes. These volunteers worked around-the-clock and on weekends to meet deadlines.

Concerns as they arose were escalated to “action items,” each assigned to responsible managers who saw the issue through to its resolution.

Overall, problems seemed to raise alternatives rather than disagreements. Management worked together, and staff experienced esprit de corps as people rallied to the demands made on them.

Reporting. Project reporting occurred weekly through ongoing management meetings and Weekly Work Package Status reports tracking items scheduled for completion.

Reviews. Two Red Team reviews took place, one in June and another in October 1992. By this point, plans had already undergone intense scrutiny in weekly management meetings. The Red Team reviews, however, brought a new perspective. They drew on technical experts from other Rockwell divisions and business units, many of whom had participated in projects at their own sites. For the most part, these experts approved the plans, suggesting only a few refinements.

Signoffs. Signoffs ensured both communication and accountability. Each manager was responsible for the work items assigned to his or her area, and handled reviews and approvals with subordinate staff. Managers reported on their progress to project management via weekly meetings.

These work items were defined in the work breakdown structure, not only by major categories and subcategories, but often to eight or even ten sublevels. Managers tracked these sublevels; when a grouping was complete, the work package was submitted, approved, and signed off by the project manager, the assistant project manager, and three members of the advisory council.

Project Evaluation. Evaluation after the project was over was planned in advance. The criteria for successful project evaluation was set up at the outset. Via the work breakdown structure, evaluation data was collected as each task was identified, assigned a number, and tracked through the project to completion.

Since each work item was assigned to a responsible manager, scheduled dates were established early in the project and updates were required at weekly management meetings, the Weekly Work Package Status reports kept management current on which projects were on/ahead/behind schedule. These reports provided an incentive to get the job done.

Testing was also built into the project, with assigned managers and dates, to ensure the system would be operational and on schedule.

Conclusion

By all accounts, the relocation project was judged a resounding success. Using a strong matrix organization and an experienced project manager, the move was planned in detail, with detailed time schedules worked out well in advance. This organization allowed the plan to be carefully refined through successive reviews by the Red Team as well as through decisions made in regular management meetings.

Staff were kept informed through periodic meetings. With support from upper management, the project manager could expect to draw on additional staff where necessary to meet deadlines.

The project itself was conducted in a climate of mixed enthusiasm and anxiety. While the project's importance generated a desire to “get the job done,” layoffs were ongoing both in Texas and California. To offset the effect of layoffs, Rockwell's use of combined staff from both sites helped create a spirit of cooperation from the start; furthermore, management continually made evident their appreciation for the cooperation of project participants.

Good project management deserves much of the credit for the project's success. Overall, the project was extraordinarily successful. The project was clearly defined and deadlines were firmly established; people came together and worked as a team; communication channels were kept open; the work breakdown schedules kept management focused on deadlines; and a strong project manager was given the authority and support needed to get the job done. The records of the project will serve the company well when another such project is planned in the future. ■

A.L. Zambrano, Ph.D., was employed at Rockwell International for six years. He has master's degrees in management, public administration, and computer resource management.

E.H. Briones is an operating systems programmer who has worked at Rockwell International for 16 years. He has master's degrees in computer resource management and management.