Quality management in a large-scale rollout project
Quality management methodology in a typical information technology project is well understood and used. Rollout projects in the information technology domain (e.g., rolling out a large number of PCs to the desktop) requires a different quality management system, one designed for logistics processes, as can be found in the manufacturing domain. Therefore, to properly cover all quality issues in a rollout project, we must integrate two different quality management systems in one and the same project.
In this paper, we'll cover the main characteristics of a large-scale PC rollout project and the quality management system we implemented and used. We'll give you details on the design and implementation of the system, as well as how we used it in the actual rollout. Finally, we'll show you some of the relevant lessons learned.
Using this information should help you understand the major differences between a typical information technology project and a rollout project, as well as how a quality management system can be designed and implemented in a rollout project.
Exhibit 1. Logistics Process Structure
Characteristics of the Project
Not all rollout projects are the same, and some may do with a “standard” quality management system. In our case, the characteristics of the project were such that an additional, different quality management system had to be designed, implemented, and used. In order to understand the reasons for applying two quality management systems, it is important to understand the main characteristics of the project.
The project scope was:
(a) To replace 36‘500 desktop PCs with state-of-the-art PCs, based on four standard hardware configurations.
(b) To have PCs ready-to-run for every user at migration time, i.e., standard software and all user-specific applications had to be preinstalled, and the PC environment (Windows NT) had to be preconfigured for every user.
(c) To rollout all PCs within a period of five months.
Besides replacing the PCs, there was one major goal that had to be achieved at all times:
In 80% of all cases, business operations were not to be interrupted for more than one hour per user when replacing the old with the new system.
The project was structured in two distinct ways:
• A standard project structure with a steering board, program management, several project managers, and their teams.
• A process-oriented structure, which focused on the logistics process (see Exhibit 1).
The major phases of the project were:
In order to roll out such a large number of PCs and considering the time constraints, all activities to take place during rollout, had to be designed in detail by specialists of the different disciplines (planning, production, staging, etc.). (Duration: Four months)
The implementation consisted of the following subphases:
• Prototype: During the prototype phase, five PCs were rolled out to verify the basic correctness of the process designs. These were then reviewed and adapted where required. (Duration: Two weeks)
• Pilot: In the pilot phase 1,000 PCs were rolled out. This phase was the most important one before the actual full-scale rollout, as most lessons were learned here. (Duration: One month)
• Review: The results of the pilot phase were analyzed in detail, and the processes optimized accordingly. (Duration: Two weeks)
• Consolidation: During consolidation, 500 PCs were rolled out based on the optimized processes (Duration: 2 weeks)
In this phase the actual rollout took place. When this phase started, the project was run like a manufacturing operation. Due to the time limitations, there was very little room to keep optimizing the logistics processes, so this had to be taken into account for quality improvements to take place. (Duration planned: Five months)
Logistics Process Structure
In order to organize the volume to be rolled out, the notion of a migration order was created. The major characteristics being:
• A maximum of 32 PCs, allowed deviation +10%. The main reason for this limitation was the number of PCs one migration team could install and migrate within the time constraints.
• All PCs of a migration designated for one location only, i.e., locations with a large number of PCs had to be broken into several migration orders.
• A maximum of 20 migration orders per working day, i.e., there were maximum 20 migration teams installing and migrating on any given day.
In total there were more than 1,300 migration orders, and each had to pass through the processes as shown in Exhibit 1, and had to be tracked accordingly.
The logistics process structure contains all relevant processes, each of which was owned by a process owner (PO). Each PO was responsible for one or more baseline points, i.e., for the time available to run the process for any given migration order.
Preparation and Planning (Customer)
Starting at X-40 (not shown in Exhibit 1), the customer started planning a migration order. Based on this first draft, the people responsible at the locations could start preparing the location and its users. At X-28, the migration order was finalized and passed on to Compaq's planning and preparation.
Preparation and Planning (Compaq)
In this process the migration order was analyzed and verified. The relevant information was then passed to the different process owners for their own planning, e.g., for the migration teams.
Based on the migration order the required hardware could be ordered at our production facility. As mentioned before, four different hardware configurations were available.
Production and Staging (Software)
The ordered PCs were produced and staged in our Erskine production facility in Scotland. As PCs had to be staged specific to a user, part of the production process was to write specific asset information into the BIOS, which would allow to relate the PC to the customer's user database for staging. The database contained all necessary software application information and configuration data for every user receiving a new PC. In order to stage the PCs for a specific end user, a staging area capable of staging at first 45 PCs in parallel was built and used. Software installation was via a network, connecting the customer's distribution servers with the factory—the staging servers. Staging was then done from the local servers based on the asset information and the user database.
Shipping and Distribution
Once produced, the PCs had be shipped to Switzerland, and distributed to the 400-plus locations all over the country. For that purpose a hub was created at a warehouse in Switzerland. Shipments from Scotland came in volumes of approx. 500 PCs, which then had to be allocated for distribution to their locations. All PCs had to be at their final destination on day X-1, at no later than 4 p.m.
On day X-1, at 4 p.m., the installation team would start unpacking the PCs, bring them to the desk of the designated user and install them. This included several steps to ensure that the PCs were working as required. At this time they were also connected to the network, so latest software updates could be downloaded during the night through the network. Existing PCs were moved to a storage area for wiping and disposal the next day.
On day X, at 7:30 a.m., the migration team would come in and run further tests, do some manually required configuration work, and then pass it to the user for acceptance.
The first application, which would automatically start before a user could use any other application, was the web-based user acceptance. Here, the user had to fill out an acceptance form and questionnaire, which we'll describe later.
Due to the nature of the customer's business, all old PCs had to be wiped, before they were allowed to leave the location for disposal.
Disposal/Recycling of Used Equipment
All old systems were transported to the hub. From here they were shipped in bulk to specialized firm for reselling.
Part of the overall project was the maintenance of the newly installed equipment. This process is still going on.
The Quality Management System
Two different quality management systems were designed and implemented:
• The project quality management system was designed to ensure quality and process improvements for the overall project.
• The rollout quality system was designed to ensure quality and process improvement for the logistics processes during the actual rollout.
Project Quality Management
Quality management for the project was based on the standard ISO-certified quality management system used in our company for systems integration projects. As such it does not differ from typical quality management systems for that type of project. There was one major addition, in that the customer had introduced a risk manager before the actual project together with Compaq had begun. This was mainly due to the great number of dependencies within the customer's organization. It clearly allowed the project to show the impact of these dependencies on the rollout, and forced a number of business units external to the project to do their tasks in time. Risk management continued for the duration of the project.
Rollout Quality Management
The rollout quality management system was designed and implemented in two phases.
1. The first phase consisted of quality checks on the baseline and other predefined quality checkpoints (i.e., key performance and quality indicators). The quality checks on the baseline were of a quantitative nature (i.e., either the baseline was kept or not) whereas the other predefined checkpoints were also of a qualitative nature (i.e., the user was happy with his system—or not).
2. As part of the actual rollout, all collected data from the quality checkpoints were regularly analyzed, and, where deemed appropriate, additional quality checkpoints were introduced into the processes.
The major point of measurement for the rollout quality was user satisfaction. Users could rate their specific migration experience with a rating from 1 (lowest/worst) to 6 (highest/best) based on several questions asked (see Phase 1: User Acceptance).
The goal was that 90% of all users would give a rating of 5 or higher.
The first application the user would see when starting up his new PC was a web-based user acceptance form/questionnaire. This consisted of several questions the user was asked, and where he or she could indicate a rating from 1 to 6 (best):
• Friendliness/Correctness of the migration team
• Correct installation of the PC
• Time lost due to the installation/migration
• Availability of peripherals and printers
• Availability of applications
• Availability of data.
The results of the user acceptance were analyzed on a daily basis. Every user giving a rating of 3 or lower was personally contacted, and asked for the reasons of the ratings given. The results of the calls were documented and passed on to the quality circle for further analysis.
The rollout process was split into 18 distinct steps, which formed the baseline:
|1. Migration order opened||X-28||Customer|
|2. Local preparation done||X-24||Customer|
|3. Target configuration HW/SW updated in user database||X-22||Customer|
|4. HW/SW Configuration freezed||X-20||Customer|
|5. Migration production order generated||X-19||Compaq|
|6. Migration production order confirmed||X-17||Compaq|
|7. Applications on production server installed||X-17||Customer|
|8. Configurations on production server checked||X-12||Customer|
|9. HW Production and SW Staging finished||X-9||Compaq|
|10. Applications on user's application server installed||X-5||Customer|
|11. Configurations on application server checked||X-5||Customer|
|12. Homedrive migration ready||X-5||Customer|
|13. Shipments arrived in central warehouse||X-3||Compaq|
|14. Migration preparations done||X-3||Customer|
|16. User acceptance||X||Customer|
|17. Migration order closed||X+5||Compaq|
|18. Quality assessed||X+6||Customer|
Each baseline step was a quality checkpoint, and as such had to be confirmed in the tracking database, either manually or automatically by system feedback. Deviations were handled as follows:
• If any given checkpoint was not confirmed on the due day at 0:00, the quality checkpoint in the database would turn to light green.
• If any given checkpoint was not confirmed on the due day, it would turn yellow.
• If any given checkpoint was late, it would turn red.
• Migration orders with any quality checkpoint being yellow or red automatically went on hold, and raised an exception with the rollout coordination managers. They together analyzed the cause of the deviation and defined the actions necessary to get back on track.
• In order to release the migration order, a change request had to be generated and approved by both rollout coordination managers. This was the only way to make changes to the baseline, once the migration order had been opened in Step 1.
All checkpoints were recorded and consolidated for later analysis. For that purpose, each checkpoint was not only tracked with a delay threshold, but also with quantitative thresholds; e.g., number of status red, number of change requests, etc.
All possible and impossible exceptions were analyzed before the actual rollout took place, and actions to correct the exception were defined. This became a reference cookbook in handling exceptions. In addition, exceptions internal to the customer were flagged in a ticketing system, and tracked by the customer's rollout coordination manager. All tickets had to be closed the same day. Exceptions internal to Compaq were flagged via an escalation procedure, which at its heart had a central web-based repository, into which all installation/migration teams could send the details of the exception. These exceptions had to be addressed as they were received. As with the baseline checkpoints, all exceptions were recorded and consolidate for later analysis, and had quantitative thresholds assigned; e.g., number of tickets, times until tickets were closed, etc.
The baseline checkpoints did not cover all aspects required. Especially, for certain areas, the indicators were too late to act upon. For that reason, additional checkpoints were introduced in the processes. The following are two examples:
• Additional hardware tests in production:
As the customer was using specialized hardware for connecting special devices, additional hardware tests were introduced and tracked.
• Staging test:
The staging process was basically fully automated. Nevertheless, PCs had to be connected to the network for the staging to take place, and had to be disconnected afterward. In order to avoid PCs being shipped without being staged, an additional staging test (i.e. has the PC been staged) was introduced into the production line.
All data from the quality checkpoints were consolidated for quantitative and qualitative analyses. Each checkpoint had a threshold value assigned to it, which was tracked. The reports were analyzed by the quality circle on a regular basis.
A quality circle was institutionalized, consisting of customer representative; i.e., the quality manager, the risk manager, the rollout coordination manager, the process owner for planning and preparation, as well as the Compaq quality manager. The main task of the quality circle was to analyze the reports containing the results of the quality checkpoints, and to define actions for improvements. These could address process flaws detected, but also holes in the system, which were not properly tracked yet. A number of actions were thus defined, e.g., doubling the capacity of the staging cell in order to prevent delays in the staging process, as the capacity of the staging cell turned to be a bottleneck when hundreds of PCs had to be staged on one day. In order to further quality with the user, trailer calls consisting of specialized questions were placed with more than 100 users after about one third of the rollout was done.
All changes to the rollout process, including changes resulting from quality checkpoint evaluation, went through a change management process, thus ensuring proper implementation while the rollout was continuing.
Due to resource constraints, some quality issues were not introduced into the logistics process while they were designed. The later introduction of quality checkpoints into the logistics process proved to be difficult and time consuming, as the logistics chain was designed in every detail and with timelines to it. Additional effort for quality tracking had a direct impact on the timeline. Some processes had to be adapted to include quality checkpoints.
At the heart of the quality management system was a database, containing mechanisms to track most of the quality checkpoints. Without electronic support, tracking would not have been possible to the extent it was done.
The differentiation between deviations which had to be handled on an immediate basis by the rollout coordination managers, and the evaluation of the consolidated results of checkpoint tracking by the quality circle for short/mid-term corrective action, proofed to be sensible. Immediately required actions typically addressed specific rollout problems, whereas the later resolved more basic process problems.
A project that involves key players from different organizations requires a quality management system also tracking the integration quality of processes, i.e., interdependencies. More often than not, a quality checkpoint revealed a process flaw in a precedent process rather than where the deviation occurred.
The implementation of a specific quality management system for the rollout was a major factor for the successful completion of the project. In fact, due to the confidence in the logistics process raised by the quality checkpoint data, the rollout was accelerated considerably and the project finished one month ahead of schedule, i.e., the actual rollout took four months to complete, rather than the five month as planned.
Proceedings of the Project Management Institute Annual Seminars & Symposium
November 1–10, 2001 • Nashville, Tenn., USA