"Flawless execution" the 1B processor project

A DOCTOR IN MINNESOTA asks a New York City hospital for the results of a patient's MRI scan. In seconds, it appears on her screen and she's able to proceed with her diagnosis. A computer company in Orlando faxes an order to Malaysia for 5 million additional memory chips. In Los Angeles, a college student facing final exams gets a call of encouragement from his father in Chicago. Although we don't always realize it, communications such as these are a critical part of our day-to-day lives and we depend on them as we become more and more of a global family.

The backbone of the AT&T switched network that makes all of this possible is the 4ESS Electronic Switching System, a large, complex, digital system that handles up to 107,000 connections and 700,000 calls per hour in real time. A total of 135 of these systems, their interconnections, and the access to them comprise the AT&T long-distance communication network that serves millions of customers each day (see Exhibit 1, following page).

The 4ESS switch and the 1A Processor that controls it had served the network well since 1976. However, increases in calling levels and call complexity eventually demanded an increase in capacity. In the late 1980s, network planners realized that the growth of basic calls, a shift toward calls that required more computer resources, more complex routing and billing schemes, and the desire to incorporate new technology would require more processor real time and memory, exceeding the availability of those resources in the 1A Processor.

Of the alternatives considered, the most feasible, from both the business and technical standpoint, was the development of a new central processor for the existing 4ESS switch: the 1B Processor. This approach had a number of advantages. The 4ESS switch is feature-rich as a result of past investment in software development. Replacing the central processor allowed this large embedded software base, as well as much of the embedded capital base, to be reused. It also provided more memory, faster real-time performance, and a means for supporting new technology in the future. Because only the central processor would be replaced, the development and deployment could be done on a schedule that would provide relief to the network before 1995, when the real-time problems and lack of sufficient memory were expected to become critical.

The Challenges. Replacing a processor in a live, fully operational environment is a formidable task. It's like performing a delicate heart transplant—there is no room for error when considering the poten tial impact to the AT&T network and the millions of customers it serves. Therefore, the most striking customer imperative was “flawless execution—with no interruption to service.” The project and the product had to be right the first time in virtually every respect.

The 1B processor, a keystone of the network that makes advanced telecommunications services possible, took four years to develop and test and a year to deploy in 135 AT&T Electronic Switching System sites. An aggressive schedule, stringent reliability requirements, difficult technical issues, and a huge, dispersed team—all these hurdles were cleared thanks to exemplary application of project management principles and techniques.

The 1B Processor Project began in 1990 with a comprehensive, fully integrated project plan that covered the details of development, manufacturing, testing, and field deployment through the closeout of the project in April 1995, when the last 4ESS switch was upgraded with the new 1B Processor. The use of established project management principles from virtually every area of the PMBOK Guide was essential to the project, while the unique challenges of size and complexity required innovative project management techniques.

Exhibit 1. The 1B Processor controls the 4ESS Electronic Switching System, a key component of the At&T switched network, which is the backbone of a telecommunications system that makes possible all the sophisticated communications services that we take for granted today. The 4ESS switch handles up to 107,000 connections and 700,000 calls per hour.

The 1B Processor Project represented one of the largest single investments in new network technology AT&T had ever made. The deployment schedule was faster than for any previous product. The reliability requirements were extremely stringent. Taken separately, each issue made the project a high risk. Taken together, they created a project that might seem to entail insurmountable obstacles. Yet the project was completed ahead of time and under budget.

Too good to be true? Despite problems along the way the single most important factor in overcoming all problems was, the commitment of the project team to the project, and their willingness to work together to keep the project on track. Here are some of the challenges they faced and mastered.

Aggressive Schedule. Generating a set of system requirements for the 1B Processor was a critical step. These requirements served both as a specification for hardware and software development, and as a guideline for product verification. In addition, they set customer expectations for 1B Processor performance. To reduce the project duration, the generation of specification documents overlapped the hardware and software development processes. Completing this initial step under intense scheduling pressure was a significant challenge. After an early prototyping phase had established feasibility, the plan for the entire project was finalized. It allocated four years to develop and test the 1B Processor, and a little over one year to deploy it in the 135 AT&T 4ESS switches. The four-year development and testing interval required concurrent engineering of hardware, software, and documentation. The deployment schedule was three times faster than those used for any previous product, with four switches being upgraded on a typical weekend. Due to the high risk, extensive installer training had to be developed, scheduled, and managed. The installation of the 1B Processor was in progress in up to 50 sites at one time across the United States. This was carefully managed using a specially developed communication system for monitoring progress and problems remotely so that corrective actions could be taken quickly and universally.

The hardware design and development of the 1B Processor had to be completed first and was in the critical path—testing, software integration, performance verification, and production could not occur without its completion. An innovative technique used to manage this involved the use of Hardware Benchmarks—milestones at which functionality and/or progress could be clearly demonstrated. These 50 benchmarks were specifically defined to closely manage the hardware portion of the schedule.

High Reliability. The reliability objective for the 1B Processor was simple: The 1B Processor should be at least as reliable as the 1A Processor. But while the 1A Processor had been tuned to its level of reliability over a 15-year period, the 1B Processor had to achieve that same level of reliability by the end of its deployment—only one year after its first application. This general objective translated into specific requirements for system, hardware, software, and procedural reliability. The intent of the requirements was to ensure that the introduction of the 1B Processor was transparent to customers. A special Reliability Assurance Team was chartered to “leave no stone unturned.” They identified reliability issues that translated to improvements in the design and in the installation procedures. Extensive testing programs for both hardware and software were executed to verify that the requirements were met. These testing programs required detailed planning, monitoring, and control in order to ensure a high-quality product.

Technical Issues. The 1B Processor was required to be “like a 1A.” Yet most of the experts who had developed the original 1A Processor were gone. Recruiting trained software staff to work on the project was difficult because the software development environment and programming languages used in the 1A environment were unique to the 4ESS switch area of AT&T. It took six months to a year to train new software developers. Special training courses were designed and developed to accelerate the learning curve.

Hardware design was made more complex by a decision to use established, proven technology to lower the reliability risk. But the technology selected restricted the designers’ choices and made it harder to achieve the technical improvements that the 1B Processor was expected to provide—the very improvement that would make the processor a good investment.

Risk Management. To minimize the high risk of deploying the 1B Processor with no service interruption, special mitigation strategies were implemented that included strong owner-project team partnering, including a shared leadership approach to the planning and implementation phases; project audits conducted to evaluate the project and processes so that potential problems and risks could be identified early enough to take corrective actions; short-duration Quality Improvement Teams focusing on specific areas that required process improvements; and comprehensive application of the PMBOK Guide Risk Management process to the deployment phase.

Project Management. The overall project involved redesigning the fully duplicated processor hardware while preserving the millions of words—each word consists of three bytes—of existing call-handling application software (operational, administrative, and maintenance). In addition, and perhaps more significantly, it involved installing and retrofitting the new 1B Processor in each live, fully operational 4ESS switch in each of the 135 4ESS switch sites over an aggressive deployment interval of only 16 months. The project can be viewed as having two major phases: development and deployment.

The intense level of activity caused by the short schedule and the rapid deployment of the 1B Processor flowed through the entire project management process, affecting each of the functional organizations in turn. Managing this project successfully required many techniques, including extensive staff training, the use of best current practices, Quality Gates, and the repeated use of formal project audits.

Dedicated teams were assigned separate functions, with individual organizations focusing on either 1B Processor development, manufacturing, testing, documentation and training, or deployment. While this helped to foster a sense of ownership for each function, it increased coordination and communication problems. These teams were new and were formed specifically for the 1B Processor Project, although they drew, as much as possible, from staff with relevant experience. The large number of dedicated, geographically separated teams could have become an insurmountable barrier if not for the strong executive support shown for this project, which manifested itself as executive advocacy, appropriate funding, and a determination to prevent inter-organizational issues from getting in the way.

The project plan established a series of Quality Gates—significant milestones spanning the entire project, from requirements to field deployment. All milestones were met—in fact, the final project review was held a number of weeks ahead of schedule.

Each of the 135 installation sites was unique, creating, in effect, 135 separate “projects” in the installation phase. As many as 50 of these subprojects were executed simultaneously during the peak of the installation interval. Additional power systems had to be designed, engineered, installed, and tested on a site-specific basis to provide power for the new 1B Processor. Differing floor plans required varying degrees of rearrangement of existing equipment just to provide sufficient physical space for the new processor near the old 1A Processor. This was required because the 1A Processor still handled live calls until after the retrofit took place. Cabling systems had to be engineered on a site-by-site basis. In some cases, doorways had to be enlarged and special ramps constructed just to transport the processor to the equipment lineup.

Field deployment of the 1B Processor was completed smoothly according to the schedule. As many as six 1B Processors were brought into service in AT&T‘s network in a single weekend, with few calls lost. So few problems were encountered that the upgrades continued even during peak holiday calling seasons during 1994-95, times when the longdistance network is normally left untouched.

The reliability data collected since the installation of the 1B Processors indicates that the reliability requirements specified in the Feature Specification Document have been exceeded. The performance of the 1B Processors now in service is better than that of the previous 1A Processors.

The People. The owner, or customer, of the 1B Processor Project and the project team were two distinct organizations within AT&T during the project's life cycle. The owner was a part of AT&T's Communications Services Group, Network Services Division, and the project team was a part of AT&T's Network Systems Group. While both were part of AT&T, their respective roles, functions, and organizations were clearly defined and distinct. The project team, along with other AT&T functions, has since become a part of Lucent Technologies.

Exhibit 2. The project team for the 1B Processor Project included about 2,800 people across numerous remote locations. That, combined with the aggressive schedule and technical complexity of the project, made a strong partnering approach a must. The owner, a distinct organization within AT&T at the time of the project, was fully integrated into the project team, participating in all meetings and decisions. The executing organization, once within AT&T‘s Network Systems Group, is now a part of Lucent Technologies.

Because of the project's size, complexity, and aggressive schedule, a strong partnering approach between the project team and the owner was taken to ensure that the owner's requirements and expectations were met or exceeded. This was done by fully integrating the owner as an equal part of the formal project team (see Exhibit 2). The owner participated as an equal partner in all meetings, reviews, problem and/or issue resolution, and decisions. Although it was uncomfortable, at first, to have the owner present at every project meeting, this was invaluable in keeping the project focused on the owner's needs and expectations.

Executives at the vice-presidential level were identified for both the project team and the owner organizations. These executives developed a close working relationship and served as the mutual project “champions” by removing inter-organizational obstacles.

The project manager with overall project management responsibility was the manager of the Corporate Product Realization Process (CPRP). The initial CPRP project management team reporting to the project manager included a technical manager and six staff members experienced in project management, with a wide variety of technical expertise in engineering, manufacturing, and installation. The owner had a parallel project management organization in its Network Services Division that focused on the quality process, office deployment scheduling, and contingency planning for office operations. Both of these managers worked very closely throughout the project planning and execution phases.

Altogether, the 1B Processor project team consisted of approximately 2,800 people from nine distinct functional organizations located in six different cities in four states. Both the size of the project team and multiple geographic locations made project management a real challenge, requiring strict adherence to the principles laid out in the knowledge areas of the PMBOK Guide, as well as innovative thinking.

Scope Management. Jointly defined by the project team and the owner, the scope of the 1B Processor Project was embodied in several documents with different levels of detail. The initial CPRP Project Plan contained a four-page Project/Product Overview that represented the high-level scope of development and deployment. This scope statement was used to guide the project team in the creation of the detailed CPRP Project Plan, which included the work breakdown structure, master schedule, staff and capital resource cost estimates, and Quality Plan. The Quality Plan section of the CPRP Project Plan further contained the detailed project deliverables, performance measures, and Quality Gate exit criteria.

In addition, the scope statement was used to develop a Feature Specification Document for the 1B Processor that defined the detailed product requirements and performance measures. The Feature Specification Document was used to guide the design and development of the processor hardware and software, to define extensive testing programs to ensure that product requirements were met, and to focus the owner's acceptance testing to ensure that product performance criteria and expectations were met or exceeded.

The product requirements and the CPRP Project Plan were instrumental in determining the overall five-year project schedule and the financial scope of the 1B Processor Project. The exact budget amount cannot be disclosed, but it included both staff and capital costs for design and development, the cost of manufacturing material and labor for the production of the processors, the capital associated with special manufacturing test facilities, and the cost of engineering and installing the processors in each of the 135 4ESS switch sites. In summary, the 1B Processor Project was a very large, multiyear, multimillion-dollar project.

After the CPRP project management team developed an executive summary of the project's scope and objectives, they asked the eight other functional organization managers to prepare a mission statement and identify that organization's customers and suppliers within the context of the project. This was presented by each functional manager at the project kickoff meeting.

At this session, representatives of the entire project team—consisting of approximately 60 managers and department heads from all nine functional organizations—met to review the project's scope and objectives, to understand the product's design, to describe each individual organization's role, to provide support for and commitment to the project, and to begin building team relationships. With the knowledge gained from the kickoff meeting, the team then developed the detailed project plan.

The plan consisted of four major sections. The Executive Summary provided team members with a common view of project scope and objectives. The Functional Overviews, nine in all, provided the specific contributions of each project function and its relationship to other functional organizations. Each of these Functional Overviews covered mission/scope, management approach, customer/supplier model, risk/contingency plans, staffing/capital profile, and change control process.

The CPRP project management function's overview, in addition to the above, included the project team meeting schedule, a status report form, action register and process, escalation process, schedule tracking/reporting process, and quality/financial review process.

The project schedule contained a detailed work breakdown structure with all tasks, activities, and milestones; task durations; and all dependencies, both within and between the project functional organizations. The final plan contained over 1,400 WBS elements. The last section of the project plan contained the Quality Plan. The project plan was a fully integrated, 440-page, living document that was used to drive the project throughout its life cycle. Each project functional organization also developed a more detailed, subtending plan for their functions that supported the integrated plan.

A change control process to address scope changes was clearly defined in the CPRP Project Plan. This process, along with dialogue and negotiation between the owner and the project team, was very effective in minimizing “scope creep.”

Symantec's Time Line project management software was used to manage, track, and report on progress of all tasks, activities, and milestones in the project plan. Reports were prepared for each functional organization prior to monthly project meetings. Updates from the meetings were integrated by the CPRP project management team and distributed one week later to the entire project team. The S-Curve derived from this information was used to assess actual project progress against the plan. In addition, an Action Register process was used for managing project issues, and an Escalation process was used for raising the level of management attention and commitment to resolving issues that could prevent the project from achieving its objectives.

Time/Schedule Management. While “flawless execution” was an imperative on the 1B Processor Project, schedule was also critically important. First, development and deployment had to provide real-time and memory resource relief to a number of 4ESS switches as early as 1994. Second, the deployment in all switches had to be completed in a short interval so that the next planned software release could provide the new services and features that AT&T had committed to delivering to its customers.

After an initial feasibility study, the project officially began in August 1990 when the product definition was complete; the preliminary sales, distribution, implementation, and support plans were defined; and the initial project team was identified. The project was defined when the preliminary CPRP Project Plan was made available. Owner approval occurred in November 1990 when all requirements documents were completed. The project was closed out in May 1995, after five years—six weeks ahead of schedule.

The 1B Processor four-year development interval included requirements, architecture, design, development, testing, and manufacturing. This work was all defined in the project WBS, and was planned and tracked using Time Line project management software. Special post-processing software was designed to produce a project cumulative S-Curve and monthly graph of baseline, projected, and actual progress directly from the Time Line files. This was the primary control system used to track, monitor, and control the schedule.

The aggressive 16-month deployment schedule included the delivery, installation, and retrofit of the new 1B Processor in each of the 135 AT&T 4ESS switch sites across the United States. Delivery was a two-week interval that was used to transport a 1B Processor to its 4ESS switch site via a dedicated, air-ride van. Installation consisted of 10 weeks from the time the processor arrived at the site to the actual retrofit. Retrofit took one night, and consisted of cutting over from the 1A to the actual operation of the 4ESS switch on the new 1B Processor—while still handling live network traffic. The first in-service retrofit took place in a small office in March 1994, followed by a controlled introduction until the retrofits were ramped up to four per weekend.

Project Cost/Resource Management. As the requirements phase of the project approached stability, the team development and planning phase was started in order to define the scope of work and the cost of successfully accomplishing it. Extensive time was invested in developing the very detailed WBS that would provide the basis for establishing schedule and costs. Once the work was defined, the managers of each project functional organization defined the time and resources, both people and facilities, required to accomplish the work. The CPRP project manager then determined how and when resources and funding would be required by using the WBS. The output of this activity was developed into a Business Case, and the results were evaluated to develop a price for the customer and to ensure that appropriate funds were available to support the project over its five-year life. The Business Case was reviewed annually with the owner.

Resources, both staff and capital, were managed against the original budget using a cost control system based on account codes linked to the WBS. The monthly data was used to assess project cost performance during formal quarterly reviews.

A cost containment strategy of the 1B Processor design was to maximize the use of existing component and assembly technologies. This allowed the manufacturing team to take advantage of existing assembly and testing facilities and experience gained from previous developments. The purchasing agents were also able to obtain volume discounts by combining the needs of this project with other projects. Since the vendors of these components were already qualified, less time and cost had to be spent in this area.

The actual project cost at completion was 5 percent below the original budget, a significant amount considering the financial scope, which was in excess of several hundred million dollars. These positive financial results were attributable to sound staff and capital estimating methods based on the detailed WBS, accurate and timely cost accounting, periodic quarterly reviews to assess overall project cost performance, competitive material procurement without compromising quality, and the rigorous use of quality techniques. The latter subsequently resulted in high product yields, very little re-work, and extremely low defective product return rates, contributing significantly to the project's bottom line.

Quality Management. The 1B Processor Project followed the two major elements of AT&T‘s Quality Policy: meeting or exceeding customer expectations, and continuous improvement. The basic philosophy is that all work is part of a process, prevention is preferred over corrective action, and quality is achieved by design.

Each process was fully documented by a process flow diagram, suppliers/inputs, outputs/customers, and in-process metrics. A formal Process Quality Management and Improvement methodology was used to manage all processes, including the CPRP project management process. During 1992–93, all project functional organizations obtained ISO 9001 Registration.

A Quality Plan was an integral part of the CPRP Project Plan. It described in detail the owner's expectations of how each project milestone should be passed and contained a matrix that defined when each quality element was due, who was responsible for its delivery, and who the customer for the deliverable was. The project team reviewed each deliverable and documented an agreed-upon description of it, ensuring that the supplier and the “customer”—either the owner or another project functional organization—had a common understanding of each deliverable.

A comprehensive set of Quality Gates was defined in the Quality Plan with deliverables, metrics, and specific exit criteria to ensure that the product and project expectations were met and that the project was on track. A total of 25 such “gates” were used that spanned roughly five years.

Quality assurance and control methods specific to the functions performed by each project functional organization were used. The Manufacturing Function, in particular, provided a striking example of the application of statistical quality assurance and control techniques and the use of quality circles.

The 1B Processor consisted of a number of new circuit pack designs. The factory was responsible for the production and extensive testing of these circuit packs to meet the highest level of product quality and reliability. Each circuit pack was uniquely bar-coded and electronically logged in and out at every operation during the assembly and test processes. Measures were categorized into 10 types, such as first pass yield, dropout rate, functional test pass rate, device defects, etc. Data was electronically collected in real time on all of these measures, and compared with upper and lower control limits to assess process performance on both a per-circuit pack code basis and on a per-operation basis. Feedback was immediate, and Pareto techniques and root cause analyses were used to address quality issues quickly.

Weekly reviews that were held to assess the data included the shop staff and shop management. If problems with a specific process or operation were identified, the shop staff was the first to be asked for their views on how to eliminate the problem(s) and improve process performance. This “quality circle” approach was very effective in addressing quality issues and in maintaining consistently high levels of process performance.

The use of a Quality Plan, Quality Gates, and Quality Assurance and Control methods enabled the project team to manage supplier performance, remove variability from the processes, detect and correct defects before they became owner-found faults, and manage risk associated with changes. Both the product and the project quality were so impressive that, in 1995, the 1B Processor Project was one of two recipients of the prestigious, annual AT&T Bell Laboratories President's Quality Award.

Human Resource Management. How do you get a team of approximately 2,800 people, organized in a weak functional matrix, from organizations located across the country, to share a common vision, truly operate as a team, and remain motivated for five years?

Shortly after the kickoff meeting, the project team's executive and the CPRP project manager traveled to each functional organization to convey the 1B Processor Project needs and expectations and to enlist the support and commitment of the team members. The entire organization's staff heard the same message. This reinforced the importance of project management, visibly empowered the CPRP project manager, and allowed a personal relationship to begin to be established.

The owner's project executive videotaped a message expressing the critical nature of the 1B Processor Project. He also stressed the need for strong teamwork to achieve “flawless execution.” This videotape was shown at a monthly CPRP project team meeting, and copies were given to each functional manager to show in their respective organizations.

The monthly CPRP project team meetings were the vehicle for establishing strong teamwork, or partnering, across all functional organizations of the project team. The project team did a good job of pre-determining how they would manage change together. When, inevitably, change occurred, there was no hiding of evidence, no cover-ups, and no finger-pointing. Instead, all team members involved in the change bonded together to define the problem quickly, analyze the cause, and resolve it. Sometimes simple one-on-one negotiation worked; sometimes the project manager handled the negotiation; and sometimes a special process management team provided an extensive effort to resolve the problem.

Early in the project, it was decided to “cross-pollinate” functions in order to build an even stronger team and eliminate potential barriers between functional organizations by sending key personnel to work in other functions. By doing so, each function gained a better appreciation of the other functional organization's role and a stronger team relationship was established. When the deployment phase was about to begin, key project team members from all organizations were invited to the first retrofit. Until then, everything had been only on paper. By being present at the actual retrofit, team members were able to observe the “real” thing. This heightened their awareness of the need for flawless execution and brought them even closer together as a team.

Many different methods were used to recognize individuals, small teams, and managers for their contributions along the five-year project journey. Celebrations were held when a major milestone was completed. Quality Improvement Teams received formal awards when significant, sustainable improvements were made to process performance. When a Quality Gate was successfully passed, the project executive took the functional managers out to dinner. Managers also provided simple “thank you's” for the many day-to-day contributions of members of the team.

The manufacturing team developed an extraordinary sense of teamwork and ownership. A “factory within a factory” was created to handle the 1B Processor and the team was thought to be so important to the project that they were taken to the nearby sites where actual 1B Processors were being installed to give them an understanding of how the processor was being used. When an executive or the owner visited the factory, team members gave tours of their work and assembly areas. When a 1B Processor was being shipped, they all signed their names to a certificate that was attached to the processor. All of this created a strong sense of pride and personal ownership in every person on the manufacturing team.

Communications Management. Communication is, at times, difficult between two people. In the case of this project, it seemed like an insurmountable task; but up-front definition of the communication approach proved extremely effective. Early on, the project team learned the importance of open and honest communications—horizontally within the project team, vertically to upper management, and with the owner. Misunderstandings, disagreements, and even hostilities will sometimes occur between people and this project was not immune to them. But because of open communications at all levels, the spirit quickly became one of “How can we meet the project needs?" instead of “We can't do that because…”

Regularly scheduled project team meetings and executive meetings were the primary communication mechanisms. Regular CPRP project team meetings were held monthly and all project functional managers or their representatives were expected to attend. The functional managers presented their WBS and schedule progress. Issues were raised, addressed, and logged in the ongoing Action Register. Quality Gate progress was also reviewed. The final project status was based on this information.

Status reports were generated monthly after the CPRP project team meetings. Each report included the overall project status (On Schedule, Alert, or Critical), a summary of recent accomplishments, and a list of project concerns and countermeasures. This report was prepared by the CPRP project manager and was reviewed by the project team executive. It was then distributed electronically to all members of the project team.

Project team executive meetings were monthly department head level meetings with the project team's executive. The focus was on “weak spots,” process deficiencies, and plans to shore up the deficiencies using Quality Improvement Teams that analyzed problems and identified countermeasures.

Quarterly owner/project executive meetings included the directors of each of the project functional organizations. They focused on overall project status, the Quality Plan, and the evaluation of progress toward the next Quality Gate.

In addition, several unique communication needs were met by using novel techniques. Interactive, satellite-based training was simultaneously provided to multiple locations across the country for installation personnel. At the end of each 15-minute module a “quiz” of five or six questions was given. A response transmitter was provided to each trainee for registering the answers electronically. The instructor received a summary almost instantaneously, and could go over areas that were not understood.

A comprehensive installation status and problem tracking system was used to communicate with the geographically-separated 4ESS switch sites and to monitor the installation progress. It provided universal, PC-based access and an easy, menu-driven format for reporting status and problems. It filled the need for effective, two-way communication to ensure that reported troubles were resolved quickly and that solutions were available to all sites. The tracking system generated various automated reports in order to make timely, useful information available to everyone who needed it.

Accepting the Project of the Year Award on behalf of Lucent Technologies at the PMI ‘96 awards ceremony in at PMI ‘96 in Boston, Mass., are (left to right) Paul Rutkowski, PMP, technical manager, Toll Switching R&D Project Management; J. Carl Hsu, vice-president, Advanced Technologies; and Stanley Golinski, manager, Product & Offer Realization, Toll Switching Voice & Signaling.

Risk Management. The collective use of project management principles and all types of tools and techniques, along with comprehensive, detailed planning and execution, were instrumental in minimizing project risk. Both internal and external risks were identified, assessed, and monitored on an almost day-by-day basis. Three major risk mitigation techniques were used: project audits, deployment risk management, and an office contingency plan.

Extensive and repeated use was made of project audits that consisted of an evaluation of some part, or parts, of the project by internal and external experts independent of the project team. The purpose of these audits was to assess the adequacy of project plans, the effectiveness of the project management systems, and the adherence to project policies, procedures, and guidelines. The owner participated in every audit to ensure that the owner's needs, expectations, and requirements were being met. In each case, corrective actions were identified that further strengthened the project, such as adding critically needed experience, expanding testing facilities, and developing an online system to communicate with the geographically separated installation teams on a site-by-site basis.

A formal risk management approach, based on the PMBOK Guide model, was used during deployment. A cross-functional team used the model to define a more detailed process consisting of the objective, inputs, key activities, tools and techniques, and outputs for each process step—identification, assessment, and response—specifically for this project. The team then applied the process to the detailed tasks and activities of the delivery installation, and retrofit intervals included in the deployment of each 1B Processor.

There were two important risk considerations: service and schedule. Both of these were assessed individually and then expert judgment was used to arrive at a composite, overall assessment. Several remaining high-risk items were identified such as use of “old” versions of installation procedures; physical activity close to live, operational equipment; and change activity during installation. Adequate steps were then identified and taken by the project team to mitigate these risks. Residual risks and new risks were then reviewed on an ongoing basis.

The project team documented a contingency plan that allowed for reverting back to the old 1A Processor if necessary. Extensive, detailed planning was done to ensure that 1A office data was kept current and that the 1A Processor could recover after a period of inactivity. The process was clearly defined, personnel were trained in its use, and the decision criteria to invoke it or not were clearly defined. Fortunately, the contingency plan was never needed.

Project Contract/Procurement Management. The contract process for the 1B Processor Project was a straightforward fixed-price contract between the owner and the project team. The contract defined a series of progress payments averaging three per year over the five-year life of the project. The payments varied, but they were always multimillion-dollar amounts. Once the development phase was completed, the customer also paid a fixed amount for each 1B Processor when its installation was completed during the deployment phase.

The area of potential conflict with this contract arrangement was whether or not all work had been successfully completed before a progress payment was made. To reduce confusion and conflict in this area, very definitive measures, including detailed descriptions of the owner's expectations, were included in the project Quality Plan. The effort invested in negotiating a set of deliverables and pay points when the project began paid huge dividends in work efficiency by eliminating disputes.

The project plan established 25 planned review points—Checkpoints, Quality Gates and Financial Reviews—from July 1990 through July 1995. Fourteen of these were identified as pay points. The project successfully passed every pay point, with each progress payment being made in full. Some exceptions identified at several interim points were documented with recovery plans approved by the owner, and the exceptions were quickly closed to the owner's satisfaction.

Each processor consisted of over 9,700 individual parts and cables; yet the production schedule was never delayed due to material shortages or quality issues. How was this accomplished?

Procurement of all material required to manufacture the 1B Processors was managed by the assembly factory in Columbus, Ohio. A material planning team developed material plans, set estimates, reviewed material levels, and expedited orders when necessary. A supplier management team initiated and managed vendor relationships, solicited quotes, negotiated terms and conditions, price, delivery, etc., and issued the actual orders.

When all of the vendors were qualified and selected—120 in all—they were invited to the assembly factory to participate in an information seminar describing the importance of the delivery dates and the quality standards defined for the project. The vendors were introduced to the special expectations that the project required, which included providing detailed quality control results and providing root cause analysis on all defects. By mutual agreement, if a vendor felt that they could not deliver against these expectations, they were not included in the final list of qualified vendors.

AT THE END of the deployment interval, all 135 of the 4ESS switches had been successfully upgraded to the new, more powerful 1B Processor—each in a live, fully operational environment. To the uninformed, this may not sound like an extraordinary accomplishment. In reality, however, each individual processor upgrade was like a delicate heart transplant operation. Anything less than flawless execution of all of the “operations” could have created a serious network incident that would have tarnished the track record of reliability that is so important to AT&T and its customers. In addition, had the new processor not been exhaustively tested, an unusual network condition could have triggered a service problem after the first few upgrades. None of these possibilities came to pass.

From the standpoint of AT&T's customers, all of what took place was a non-event. Precisely for that reason, within Lucent Technologies and AT&T, it was a big event indeed.

Acknowledgments

The success of the 1B Processor Project is attributable to many, many people at both Lucent Technologies and AT&T. Their shared vision, dedication, and exceptional efforts made it happen and they created a project legacy that will live for many years to come.

Note

More information on the 1B Processor Project is available in Paul Rutkowski's paper, “Leading the Way to Flawless Execution,” PMI ‘94 Proceedings, and in the May/June 1995 issue of the AT&T Technical Journal,Vol. 74, No. 3, which devoted the entire issue to the project. ■

Paul J. Rutkowski, PMP, has over 27 years experience in the telecommunications field. He is manager of the project management group in the Toll Switching R&D organization at Lucent Technologies. He also serves as president of the Midwest PMI Chapter.

Stanley Golinski, PMP, is manager of project management in the Product and Offer Realization department at Lucent Technologies. He has over 31 years experience in the telecommunications field and holds two patents for the design of manufacturing facilities.